Pull from live json feed instead of static github version

[paulirish]

http://caniuse.com/jsonp.php I imagine is updated more frequently than when @Fyrd pushes it to his repo.

That true, alexis?

[Fyrd]

Yes, that's right. Updated every time I make a change to the data.

[drublic]

Hee,

what do you think about a cron job that curls the file from caniuse.com/jsonp.php, writes the new file, commits and pushes automatically. Or maybe better, on each request we check the age of our data.php and if it's older then a day, we replace it with the version hosted at caniuse.com.

Drawbacks?

[paulirish]

the latter sounds great

[drublic]

I've created a new branch for this to not pollute my master-branch with more unmerged commits. You can find it over here.

I am requesting the file from http://caniuse.com/jsonp.php, remove the parenthesis as it's no callback-function, write the contents into the new file and that's about it.
I've also added a small logger, which writes into a file, each time the jsonp.php-file is requested.

It looks like we don't need the data.json file anymore, as we can write the requested data from caniuse directly into the cache. This is a bit more performant then opening files twice, I guess.
Maybe I'll look into it later.

cc: @jonathantneal, @nimbupani

[jonathantneal]

I have a cron job running on my wikies server that does this, except it might be putting from the original github repo.

So ... uh ... obviously I'm putting my +1 on this! Of course it would make sense to have the cron job run not off my sandbox site :D

[drublic]

I think including the cron into the script adds a bit more independence to the repo.
On my local machine I have no cron which requests the latest version of caniuse. We could avoid errors for everyone dealing with the repo by including this fix once.

[raucao]

What's the status on this? Unfortunately, the current data is quite outdated at the moment. I wanted to use it for getUserMedia, but all the current browsers are marked as unknown, while updated on caniuse.com.

Should we update manually in the meantime? It's not much work, when you discover something outdated, and I don't think the idea of being so dependent on a single person's data, awesome as he or she might be, is a good one.

[raucao]

One more thought: wouldn't it be a preferable situation if @Fyrd would pull data from a GitHub repo for caniuse (where we can all send pull requests) instead of only occasionally pushing it to all of us?

Sorry if I'm missing something or there was already a discussion about this. Didn't really follow these 2 projects before.

[drublic]

@skddc I've pushed my changes to a new branch "caniuse-data". Check out this one if you want to use an automatic pull from caniuse.com/jsonp.php.

FWIW the problem with getUserMedia is that caniuse.com/jsonp.php only returns "old" data. I've manually included the latest version of jsonp.php in my local repo. Didn't change anything.
Sorry, but I guess we'll have to wait until @Fyrd updates the file.

[raucao]

You see the absurdity of having to wait for someone to update a file, when we're all using GitHub, and it's really just about 3 characters that need to be replaced, don't you?

[drublic]

Please feel free to submit a PR which pulls more recent data from the source you can find.
As we build upon caniuse we are depending on the data provided. Sorry for any troubles.

[jonathantneal]

The most recent source is always the one on his website? I'll make a cronjob to pull from that right now.

[drublic]

@jonathantneal I thought this is the most recent. I've implemented a kinda "pull cron" in the caniuse-data-branch. It's not a cron though :)

[jonathantneal]

@drublic , I tried writing a PHP curl script ( source @ https://gist.github.com/2293056 ) that pulled from http://caniuse.com/jsonp.php, but it always timed out. I'm stuck there at the moment.

[drublic]

@jonathantneal Did you have a look at what I did? I've implemented this with get_file_contents here. This does not time out for me.

[jonathantneal]

@drublic, I've fixed my pull from fyrd's github repo and updated this repo with his changes. That resolves a lot of the problems we were having. However, that still doesn't address the direction we want to take on this ticket.

Who updates the live site?

[nimbupani]

The live site is updated from the master every 10 mins.

[drublic]

@skddc The data for getUserMedia can now be found here.

@jonathantneal Thanks for fixing this. Which direction do you mean? As I pointed out my fixes pull the jsonp.php-file from caniuse.com and save the new version if it's older than a day. For me this is fixes the issue.
Maybe I just don't get the point…

[raucao]

@drublic What does that mean? It's still not live on the main API endpoint, right?

[raucao]

The question is still unanswered if it's enough to just complete the ticket by its original name, or if we can/should find a better solution that doesn't rely on a single person to update the data. I'd gladly help keeping it up-to-date, and during our work on @5apps we have to deal with this every day anyway. Would be great if @Fyrd could share his view on the matter.

[drublic]

@skddc No, it's not live yet or merged into the master. I want to await the "go" from @jonathantneal, because he has another solution / current state.

It feels like @Fyrd is updating the jsonp-file regularly. So requesting this file seems the best solution to me. That's why I implemented it.

[paulirish]

Yes the jsonp file gets updated as the site does. It's the best source for the data.

+1 on site pulling from here every 10minutes.

+1 on waiting for the Go from jon neal.

Hans, let me know if you need access to logs once this is in, or anything.

[Fyrd]

Yeah, what Paul said. The jsonp file is updated as part of updating any data-related part of the site.

[jonathantneal]

I am in favor of pulling from WHEREVER the jsonp is first updated. :) From the website is fine, and if it changes later, that's fine too. It's plain sense to me.

[drublic]

I've changed the time for which the file is cached on our server to 10 minutes as Divya's cron is doing and Paul +1ed.

As I think we are not really finding a solution for this at the moment, I will just keep the stuff I did in the caniuse-data branch. Maybe it comes in handy when discuss this again in the future :)