-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Report JSON parsing errors when generating CSV #39
Comments
Thanks for reporting this. Is it a large JSON file? Can you share it with me at [email protected]? |
Sure sir, thank you for looking into this. I will share it with you. I put a 20MB text file in the Hydrator, and in return I got 2.8GB JSONl file. |
Hello @edsu , just sent that json file which lead me to this error. Thank you & your team for making this tool! |
I got it. It's interesting, of the 621,567 lines there appears to be one line (307,400) that has invalid JSON on it. It looks truncated in some way. Do you remember if you happened to have a storage problem or shutdown/stop the Hydrator about midway through hydrating this tweet id dataset? It shouldn't be a problem to start/restart but I should test it out to make sure. Have you been able to hydrate successfully before? |
@opendatasurgeon could you share your ID file with me too if it's not too much trouble? This should hopefully help me figure out what might have happened. |
No sir, I got no warning or note while hydrating. It was after when I hit the CSV button, json output gave no error. Also my computer didn't sleep, nor I have a storage issue. I have ~280 GB left on the drive where I ran this program. I didn't stop the hydration, I just waited until I get an option to export. It was a continuous process on my end. Only thing missing hardware wise is a dedicated audio device. Whenever I need to hear something, I just put a headset. |
Sure Mr. Summers. Will email the ID file which caused this error and also give you the id file and the json file where I was able to get an output without an error. |
Mr. Summers, sent you an email containing id files. Please let me know if there is anything else you need. Appreciate your help again! |
Thanks @opendatasurgeon. I'm confused about the relationship between the |
Will do sir. I unfortunately did';t capture the screenshot if the % lost. I will however re-run this. And report it to you. Thank you :) |
Oh yes, before I forget to tell you. The source of the data I am using to hydrate is here: https://github.com/echen102/COVID-19-TweetIDs |
Hello @edsu Just finished hydrating the text file. No error this time. Same computer, same process (meaning continuous hydration, no-computer sleeping, no disconnects or starting or stopping), I don't get it. The CSV and JSON files are surely smaller in size. Please see the screenshots below. |
@opendatasurgeon hydration worked for me too, 53% of the tweets have been deleted! That is a shockingly large percentage for such a recent set of tweets (just over a month old). I guess it's not surprising given that these appear to be COVID-19 related tweets, and there have been widespread disinformation campaigns about it. Do you think it's possible you may have overwritten an output file accidentally when hydrating two files at the same time? I'm going to leave this ticket open because Hydrator should report an error in the JSON rather than throwing an exception. |
I closed this by accident and want to keep it open until Hydrator reports errors better. |
I don't think I did overwrite any of my file sir. I always name my files differently, and I believe I just merged the files, that were chunk of same days, together into one. I in fact left duplicates IDs in these files because I was going to clean CSV files anyways. So don't know what could have happened. Do you think maybe there is limit a I should test with a May file? I know for sure first time around the deletion % wasn't above 30-40%. Let me report it you. Thanks again for helping out Mr Summers!! |
I am not getting this error these days even when hydrating >1500000 twitter ids text file. I will keep you in loop Mr. Summers. Thank you! |
Well that's a relief. If you notice it again and can figure out a way to reproduce please let us know. I will leave this ticket open until the app reports the JSON parse error better. |
Hello, I have received this error back to back when exporting to csv file:
The text was updated successfully, but these errors were encountered: