Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report JSON parsing errors when generating CSV #39

Open
mihirp161 opened this issue May 10, 2020 · 18 comments
Open

Report JSON parsing errors when generating CSV #39

mihirp161 opened this issue May 10, 2020 · 18 comments

Comments

@mihirp161
Copy link

mihirp161 commented May 10, 2020

Hello, I have received this error back to back when exporting to csv file:
Inkederror_LI

@edsu
Copy link
Member

edsu commented May 10, 2020

Thanks for reporting this. Is it a large JSON file? Can you share it with me at [email protected]?

@mihirp161
Copy link
Author

mihirp161 commented May 10, 2020

Sure sir, thank you for looking into this. I will share it with you. I put a 20MB text file in the Hydrator, and in return I got 2.8GB JSONl file.

@mihirp161
Copy link
Author

Hello @edsu , just sent that json file which lead me to this error. Thank you & your team for making this tool!

@edsu
Copy link
Member

edsu commented May 11, 2020

I got it. It's interesting, of the 621,567 lines there appears to be one line (307,400) that has invalid JSON on it. It looks truncated in some way. Do you remember if you happened to have a storage problem or shutdown/stop the Hydrator about midway through hydrating this tweet id dataset? It shouldn't be a problem to start/restart but I should test it out to make sure.

Have you been able to hydrate successfully before?

@edsu
Copy link
Member

edsu commented May 11, 2020

@opendatasurgeon could you share your ID file with me too if it's not too much trouble? This should hopefully help me figure out what might have happened.

@mihirp161
Copy link
Author

mihirp161 commented May 11, 2020

I got it. It's interesting, of the 621,567 lines there appears to be one line (307,400) that has invalid JSON on it. It looks truncated in some way. Do you remember if you happened to have a storage problem or shutdown/stop the Hydrator about midway through hydrating this tweet id dataset? It shouldn't be a problem to start/restart but I should test it out to make sure.

Have you been able to hydrate successfully before?

No sir, I got no warning or note while hydrating. It was after when I hit the CSV button, json output gave no error. Also my computer didn't sleep, nor I have a storage issue. I have ~280 GB left on the drive where I ran this program. I didn't stop the hydration, I just waited until I get an option to export. It was a continuous process on my end. Only thing missing hardware wise is a dedicated audio device. Whenever I need to hear something, I just put a headset.

@mihirp161
Copy link
Author

@opendatasurgeon could you share your ID file with me too if it's not too much trouble? This should hopefully help me figure out what might have happened.

Sure Mr. Summers. Will email the ID file which caused this error and also give you the id file and the json file where I was able to get an output without an error.

@mihirp161
Copy link
Author

Mr. Summers, sent you an email containing id files. Please let me know if there is anything else you need. Appreciate your help again!

@edsu
Copy link
Member

edsu commented May 11, 2020

Ok, I will take a look. I rehydrated the tweet ids and also have tested generating CSV from hydrated data that I tried start/stopping, quitting and stopping the network during and wasn't able to get a corrupted jsonl file.

Screen Shot 2020-05-11 at 2 55 12 PM

@edsu
Copy link
Member

edsu commented May 11, 2020

Thanks @opendatasurgeon. I'm confused about the relationship between the output-2020-03-22.jsonl file you sent me (which has 621,567 lines) and the output-2020-03-22.txt file you sent (which has 1,069.472 lines). I would expect the numbers to be much closer together unless a large number had been deleted? Can you see the % deleted value on the dataset detail view? It looks like it didn't finish hydrating, or a significant chunk of the file was lost? Can you please try hydrating it again and see if you have the same problem? I can also try hydrating here and see what happens.

@mihirp161
Copy link
Author

mihirp161 commented May 11, 2020

Will do sir. I unfortunately did';t capture the screenshot if the % lost. I will however re-run this. And report it to you. Thank you :)

@mihirp161
Copy link
Author

mihirp161 commented May 11, 2020

Oh yes, before I forget to tell you. The source of the data I am using to hydrate is here: https://github.com/echen102/COVID-19-TweetIDs
This text file (output-2020-03-22.txt) containing twitter IDs is from March (2020-03) directory. You will see name difference because I combined all the chunks pertaining to same days together in one text file. I believe they are separate in the original data source, because GitHub only allows 25mb uploads, that's why they split each giant texts files in parts. But I am re-running it now.

@mihirp161
Copy link
Author

Hello @edsu Just finished hydrating the text file. No error this time. Same computer, same process (meaning continuous hydration, no-computer sleeping, no disconnects or starting or stopping), I don't get it. The CSV and JSON files are surely smaller in size. Please see the screenshots below.

3
2
1

@edsu
Copy link
Member

edsu commented May 12, 2020

@opendatasurgeon hydration worked for me too, 53% of the tweets have been deleted! That is a shockingly large percentage for such a recent set of tweets (just over a month old). I guess it's not surprising given that these appear to be COVID-19 related tweets, and there have been widespread disinformation campaigns about it.

Screen Shot 2020-05-12 at 8 31 24 AM

Do you think it's possible you may have overwritten an output file accidentally when hydrating two files at the same time?

I'm going to leave this ticket open because Hydrator should report an error in the JSON rather than throwing an exception.

@edsu edsu closed this as completed May 12, 2020
@edsu edsu reopened this May 12, 2020
@edsu
Copy link
Member

edsu commented May 12, 2020

I closed this by accident and want to keep it open until Hydrator reports errors better.

@mihirp161
Copy link
Author

Do you think it's possible you may have overwritten an output file accidentally when hydrating two files at the same time?

I don't think I did overwrite any of my file sir. I always name my files differently, and I believe I just merged the files, that were chunk of same days, together into one. I in fact left duplicates IDs in these files because I was going to clean CSV files anyways. So don't know what could have happened.

Do you think maybe there is limit a I should test with a May file? I know for sure first time around the deletion % wasn't above 30-40%. Let me report it you. Thanks again for helping out Mr Summers!!

@mihirp161
Copy link
Author

I am not getting this error these days even when hydrating >1500000 twitter ids text file. I will keep you in loop Mr. Summers. Thank you!

@edsu
Copy link
Member

edsu commented May 18, 2020

Well that's a relief. If you notice it again and can figure out a way to reproduce please let us know. I will leave this ticket open until the app reports the JSON parse error better.

@edsu edsu changed the title JS error at the main process, unexpected token at JSON in position 0 Report JSON parsing errors when generating CSV Sep 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants