{"id":6026,"date":"2012-07-04T11:19:55","date_gmt":"2012-07-04T05:49:55","guid":{"rendered":"https:\/\/2thenew.today\/blog\/?p=6026"},"modified":"2016-12-19T15:30:08","modified_gmt":"2016-12-19T10:00:08","slug":"export-very-large-data-from-a-mongodb-collection-using-script","status":"publish","type":"post","link":"https:\/\/2thenew.today\/blog\/export-very-large-data-from-a-mongodb-collection-using-script\/","title":{"rendered":"Export very large data from a MongoDB collection using script"},"content":{"rendered":"

In this post, we will learn how to use javascript as MongoDB shell script to export (fields of) a collection.<\/p>\n

 <\/p>\n

If you just want to execute MongoDB commands then look at the another post there.<\/a>.<\/p>\n

 <\/p>\n

OK, so let’s focus back to the objective of this post.
\nGenerally, to export data from a MongoDB collection<\/strong>, we use the following command –
\n[shell]
\nmongoexport -vvvv –host 127.0.0.1 –db my-db –username=’usr’ –password=’pwd’ –collection profiles –csv –out \/home\/data\/usernames.csv -f ‘username’
\n[\/shell]<\/p>\n

Definitely I prefer the above mentioned way. But let’s consider that you have millions or billions of records in a MongoDB collection. And you want to export all these.
\nProbably, this will just hang. Well, I am not sure what happens on your computer but on my machine it was just stuck for hours and then I terminated the process. And came up with scripting solution – where I created multiple files (chunks) of the exported data.<\/p>\n

 <\/p>\n

[javascript]
\nvar size = 1000000;
\nvar maxCount = 1;
\nfor (x=0;x<maxCount;x=x+1)
\n{
\n var recToSkip = x*size;
\n db.profiles.find().skip(recToSkip).limit(size).forEach( function(record){
\n var username = record.username;
\n print(record.username);
\n })
\n}<\/p>\n

[\/javascript]<\/p>\n

And ofcourse to run the script (if above javascript is in file named – ‘usernames.js’)
\n[shell]
\n mongo 127.0.0.1\/my-db –username=’username’ –password=’pwd’ export-usernames.js > output.txt
\n[\/shell]<\/p>\n

The above shell command will render all the output (from export-usernames.js) to the output.txt file. And mind it, that this is single file where you get all exported data. But if you want to chunkify this – just write your wrapper script or run multiple times. It’s upto you.<\/p>\n

 <\/p>\n

I hope this might help someone. Or please feel free to add your comments.<\/p>\n

 <\/p>\n

Cheers!
\nSalil Kalia
\nSalil [at] IntelliGrape [dot] com
\nTwitter LinkedIn<\/p>\n","protected":false},"excerpt":{"rendered":"

In this post, we will learn how to use javascript as MongoDB shell script to export (fields of) a collection.   If you just want to execute MongoDB commands then look at the another post there..   OK, so let’s focus back to the objective of this post. Generally, to export data from a MongoDB […]<\/p>\n","protected":false},"author":17,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":45,"footnotes":""},"categories":[1],"tags":[55,260,4846,453],"class_list":["post-6026","post","type-post","status-publish","format-standard","hentry","category-technology","tag-javascript","tag-linux","tag-mongodb","tag-shell"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t