[Moses-support] reg. moses installation

Discussion:

ram anirudh cherukupally

2021-04-22 05:39:26 UTC

Dear Moses-Support,

Please find the build.log.gz attached as per the instructions when the
build failed.
My system specs:

OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2

Command used for compiling moses: ./bjam -j4

Thanks and Regards

--
CH Ram Anirudh

Hieu Hoang

2021-04-22 07:45:09 UTC

Permalink

there seems to be a problem with the boost library. Is the disk full?
Perhaps you should re-install boost

Post by ram anirudh cherukupally
Dear Moses-Support,
Please find the build.log.gz attached as per the instructions when the
build failed.
OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2
Command used for compiling moses: ./bjam -j4
Thanks and Regards
--
CH Ram Anirudh
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
http://statmt.org/hieu

Hieu Hoang

2021-04-22 16:13:36 UTC

Permalink

i've just successfully compiled moses-4.0 on ubuntu 20.04 with boost 1.71.

there's something wrong with your boost installation

There is atleast 60 GB space, so I think it is not space issue. Has
moses-4.0 been tested for compilation using boost 1.71? Do you
recommend using boost 1.64Â (as exemplified in Moses manual?)
Thank you
there seems to be a problem with the boost library. Is the disk
full? Perhaps you should re-install boost

Post by ram anirudh cherukupally
Dear Moses-Support,
Please find the build.log.gz attached as per the instructions
when the build failed.
OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2
Command used for compiling moses: ./bjam -j4
Thanks and Regards
--
CH Ram Anirudh
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>

--
Hieu Hoang
http://statmt.org/hieu <http://statmt.org/hieu>
--
CH Ram Anirudh

--
Hieu Hoang
http://statmt.org/hieu

Namrata Hadimani

2021-04-22 17:32:50 UTC

Permalink

Hi Hieu,

Thanks for the help, I am able to successfully compile the moses ToolKit.

Regards,
Namrata Hadimani

Post by Hieu Hoang
i've just successfully compiled moses-4.0 on ubuntu 20.04 with boost 1.71.
there's something wrong with your boost installation
There is atleast 60 GB space, so I think it is not space issue. Has
moses-4.0 been tested for compilation using boost 1.71? Do you recommend
using boost 1.64 (as exemplified in Moses manual?)
Thank you

Post by Hieu Hoang
there seems to be a problem with the boost library. Is the disk full?
Perhaps you should re-install boost
Dear Moses-Support,
Please find the build.log.gz attached as per the instructions when the
build failed.
OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2
Command used for compiling moses: ./bjam -j4
Thanks and Regards
--
CH Ram Anirudh
_______________________________________________
--
Hieu Hoanghttp://statmt.org/hieu

--
CH Ram Anirudh
--
Hieu Hoanghttp://statmt.org/hieu
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support

Namrata Hadimani

2021-04-22 20:50:13 UTC

Permalink

Hi All,

I am trying to perform Language Model Training using the below command

~/mosesdecoder/bin/lmplz -o 3
<~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en

followed similar to the link below :
http://www.statmt.org/moses/?n=Moses.Baseline

But I faced this issue :
=== 1/5 Counting and sorting n-grams ===
Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
tcmalloc: large alloc 3135389696 bytes == 0x144e000 @
tcmalloc: large alloc 10451279872 bytes == 0xbc272000 @
Unigram tokens 0 types 3
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:36 2:4734547456 3:8877277184
tcmalloc: large alloc 8877277184 bytes == 0x144e000 @
tcmalloc: large alloc 4734550016 bytes == 0x32ba4e000 @
terminate called after throwing an instance of
'lm::builder::BadDiscountException'
what(): /home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53 in
void lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j] ==
0'.
Could not calculate Kneser-Ney discounts for 1-grams with adjusted count 2
because we didn't observe any 1-grams with adjusted count 1; Is this small
or artificial data?
Try deduplicating the input. To override this error for e.g. a class-based
model, rerun with --discount_fallback

So I tried changing my command to the following as :
~/mosesdecoder/bin/lmplz -o 3 <~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en *--**discount_fallback*

After this When i run the command : ~/mosesdecoder/bin/build_binary \

news-commentary-v8.fr-en.arpa.en \
news-commentary-v8.fr-en.blm.en

I am facing error like: lm/vocab.cc:324 in void
lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*)
threw SpecialWordMissingException.
The ARPA file is missing </s> and the model is configured to reject these
models. Run build_binary -s to disable this check. Byte: 66
ERROR

Could anyone help me out with this, please.

Regards,
Namrata Hadimani

Post by Namrata Hadimani
Hi Hieu,
Thanks for the help, I am able to successfully compile the moses ToolKit.
Regards,
Namrata Hadimani

--
CH Ram Anirudh
--
Hieu Hoanghttp://statmt.org/hieu
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support

Kenneth Heafield

2021-04-22 22:33:43 UTC

Permalink

Your training corpus is empty.

cat ~/corpus/news-commentary-v8.fr-en.true.en

Post by Namrata Hadimani
Hi All,
I am trying to perform Language Model Training using the below command
~/mosesdecoder/bin/lmplz -o 3 <~/corpus/news-commentary-v8.fr-en.true.en > news-commentary-v8.fr-en.arpa.en
http://www.statmt.org/moses/?n=Moses.Baseline
<http://www.statmt.org/moses/?n=Moses.Baseline>
=== 1/5 Counting and sorting n-grams ===
Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
Unigram tokens 0 types 3
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:36 2:4734547456 3:8877277184
terminate called after throwing an instance of
'lm::builder::BadDiscountException'
what(): /home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53
in void
lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j]
== 0'.
Could not calculate Kneser-Ney discounts for 1-grams with adjusted count
2 because we didn't observe any 1-grams with adjusted count 1; Is this
small or artificial data?
Try deduplicating the input. To override this error for e.g. a
class-based model, rerun with --discount_fallback
So I tried changing my command to the following as
: ~/mosesdecoder/bin/lmplz -o 3
<~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en *--**discount_fallback*
After this When i run the command : ~/mosesdecoder/bin/build_binary \
news-commentary-v8.fr-en.arpa.en \
news-commentary-v8.fr-en.blm.en
I am facing error like: lm/vocab.cc:324 in void
lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*)
threw SpecialWordMissingException.
The ARPA file is missing </s> and the model is configured to reject
these models. Run build_binary -s to disable this check. Byte: 66
ERROR
Could anyone help me out with this, please.
Regards,
Namrata Hadimani
On Thu, 22 Apr 2021 at 18:32, Namrata Hadimani
Hi Hieu,
Thanks for the help, I am able to successfully compile the moses ToolKit.
Regards,
Namrata Hadimani
i've just successfully compiled moses-4.0 on ubuntu 20.04 with boost 1.71.
there's something wrong with your boost installation

There is atleast 60 GB space, so I think it is not space
issue. Has moses-4.0 been tested for compilation using boost
1.71? Do you recommend using boost 1.64 (as exemplified in
Moses manual?)
Thank you
On Thu, Apr 22, 2021 at 1:15 PM Hieu Hoang
there seems to be a problem with the boost library. Is the
disk full? Perhaps you should re-install boost

Post by ram anirudh cherukupally
Dear Moses-Support,
Please find the build.log.gz attached as per the
instructions when the build failed.
OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2
Command used for compiling moses: ./bjam -j4
Thanks and Regards
--
CH Ram Anirudh
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>

--
Hieu Hoang
http://statmt.org/hieu <http://statmt.org/hieu>
--
CH Ram Anirudh

--
Hieu Hoang
http://statmt.org/hieu <http://statmt.org/hieu>
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support
<http://mailman.mit.edu/mailman/listinfo/moses-support>
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support

Namrata Hadimani

2021-04-22 23:40:30 UTC

Permalink

++

---------- Forwarded message ---------
From: Namrata Hadimani <***@mycit.ie>
Date: Fri, 23 Apr 2021 at 00:39
Subject: Re: [Moses-support] reg. moses installation
To: Kenneth Heafield <***@kheafield.com>

Hi Kenneth,

Actually I am facing error in creating the Tokenisation itself. I tried to
run through the tokeniser.perl script but it didn't help me . There is no
log generated for this functionality.
Could you guide me more.

*Below are the scripts I ran :*

~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en \
< ~/corpus/training/news-commentary-v8.fr-en.en \

~/corpus/news-commentary-v8.fr-en.tok.en

*And I am getting this as a response :*

Tokenizer Version 1.1
Language: en
Number of threads: 1

*After this Step, the tokens are not created and the file is Empty. How
will I fix this problem?*

*Thanks in Advance*

Regards,
Namrata Hadimani

Your training corpus is empty.
cat ~/corpus/news-commentary-v8.fr-en.true.en

Post by Namrata Hadimani
Hi All,
I am trying to perform Language Model Training using the below command
~/mosesdecoder/bin/lmplz -o 3

<~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en

Post by Namrata Hadimani
http://www.statmt.org/moses/?n=Moses.Baseline
<http://www.statmt.org/moses/?n=Moses.Baseline>
=== 1/5 Counting and sorting n-grams ===
Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en

----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

Post by Namrata Hadimani
Unigram tokens 0 types 3
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:36 2:4734547456 3:8877277184
terminate called after throwing an instance of
'lm::builder::BadDiscountException'
what(): /home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53
in void
lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j]
== 0'.
Could not calculate Kneser-Ney discounts for 1-grams with adjusted count
2 because we didn't observe any 1-grams with adjusted count 1; Is this
small or artificial data?
Try deduplicating the input. To override this error for e.g. a
class-based model, rerun with --discount_fallback
So I tried changing my command to the following as
: ~/mosesdecoder/bin/lmplz -o 3
<~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en *--**discount_fallback*
After this When i run the command : ~/mosesdecoder/bin/build_binary \
news-commentary-v8.fr-en.arpa.en \
news-commentary-v8.fr-en.blm.en
I am facing error like: lm/vocab.cc:324 in void
lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*)
threw SpecialWordMissingException.
The ARPA file is missing </s> and the model is configured to reject
these models. Run build_binary -s to disable this check. Byte: 66
ERROR
Could anyone help me out with this, please.
Regards,
Namrata Hadimani
On Thu, 22 Apr 2021 at 18:32, Namrata Hadimani
Hi Hieu,
Thanks for the help, I am able to successfully compile the moses ToolKit.
Regards,
Namrata Hadimani
i've just successfully compiled moses-4.0 on ubuntu 20.04 with
boost 1.71.
there's something wrong with your boost installation

Post by ram anirudh cherukupally
Dear Moses-Support,
Please find the build.log.gz attached as per the
instructions when the build failed.
OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2
Command used for compiling moses: ./bjam -j4
Thanks and Regards
--
CH Ram Anirudh
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support <

http://mailman.mit.edu/mailman/listinfo/moses-support>

Post by Namrata Hadimani

--
Hieu Hoang
http://statmt.org/hieu <http://statmt.org/hieu>
--
CH Ram Anirudh

_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support

Namrata Hadimani

2021-04-23 11:23:36 UTC

Permalink

Is this some installation related ?

Post by Namrata Hadimani
Hi Kenneth,
Actually I am facing error in creating the Tokenisation itself. I tried to
run through the tokeniser.perl script but it didn't help me . There is no
log generated for this functionality.
Could you guide me more.
*Below are the scripts I ran :*
~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en \
< ~/corpus/training/news-commentary-v8.fr-en.en \

~/corpus/news-commentary-v8.fr-en.tok.en

*And I am getting this as a response :*
Tokenizer Version 1.1
Language: en
Number of threads: 1
*After this Step, the tokens are not created and the file is Empty. How
will I fix this problem?*
*Thanks in Advance*
Regards,
Namrata Hadimani

Your training corpus is empty.
cat ~/corpus/news-commentary-v8.fr-en.true.en

Post by Namrata Hadimani
Hi All,
I am trying to perform Language Model Training using the below command
~/mosesdecoder/bin/lmplz -o 3

<~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en

----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

Post by ram anirudh cherukupally
Dear Moses-Support,
Please find the build.log.gz attached as per the
instructions when the build failed.
OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2
Command used for compiling moses: ./bjam -j4
Thanks and Regards
--
CH Ram Anirudh
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support <

http://mailman.mit.edu/mailman/listinfo/moses-support>

Post by Namrata Hadimani

--
Hieu Hoang
http://statmt.org/hieu <http://statmt.org/hieu>
--
CH Ram Anirudh

_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support

Hieu Hoang

2021-04-22 20:57:38 UTC

Permalink

cool. it would be good to say why the problem occurred in the first
place and how you fixed it so others can learn from it

HiÂ Hieu,
Thanks for the help, I am able to successfully compile the moses ToolKit.
Regards,
Namrata Hadimani
i've just successfully compiled moses-4.0 on ubuntu 20.04 with boost 1.71.
there's something wrong with your boost installation

There is atleast 60 GB space, so I think it is not space issue.
Has moses-4.0 been tested for compilation using boost 1.71? Do
you recommend using boost 1.64Â (as exemplified in Moses manual?)
Thank you
there seems to be a problem with the boost library. Is the
disk full? Perhaps you should re-install boost

Post by ram anirudh cherukupally
Dear Moses-Support,
Please find the build.log.gz attached as per the
instructions when the build failed.
OS: Ubuntu 20.04
RAM: 8 GB
libboost-dev version: 1.71.0.0ubuntu2
Command used for compiling moses: ./bjam -j4
Thanks and Regards
--
CH Ram Anirudh
_______________________________________________
Moses-support mailing list
http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>

--
Hieu Hoang
http://statmt.org/hieu <http://statmt.org/hieu>
--
CH Ram Anirudh

--
Hieu Hoang
http://statmt.org/hieu