3.0.0: Move to using GraphQL

[matthewmueller]

I definitely think gdom has showed us the way forward:

• https://github.com/syrusakbary/gdom
• http://gdom.graphene-python.org/

Here's a basic node implementation:

• https://github.com/IamNotUrKitty/gdom-node

There's some improvements we can make by allowing you to swap out drivers easily (http, phantom, electron) and also around pagination.

I'm not sure I'll have time to work on this without community support, but I'd also love for someone to own this transition :-)

[jspri]

Looks like a good idea!

One of the best features of x-ray is the ability to have the nested selectors crawl to different pages. Is there a provision in gdom for this?

[0xgeert]

@matthewmueller any reason why this graphQl-ish way of doing things would be better suited for expressing crawlers than what x-ray does atm? Is there anything lacking in expressiveness currently?

[matthewmueller]

@gebrits expressivity-wise i don't think it would add too much, but it would greatly simplify the internals of the library, put us on a more standard course and provide better error handling.

[matthewmueller]

@Crazometer yep, you can visit within a page to another page.

[rchipka]

Looks similar to the way Osmosis is doing things.

IMHO it seems more intuitive to compose the data structure in code rather than in a string. Probably makes extending and debugging easier too.

Here's the Osmosis equivalent of the example:

osmosis('http://news.ycombinator.com')
  .set({
    items:
      osmosis.find('tr.athing')
       .set({
          rank:    'td span.rank',
          title:   'td.title a',
          sitebit: 'span.comhead a',
          url:     'td.title a@href',
          attrs: {
             score:    'span.score',
             user:     'a:eq(0)',
             comments: 'a:eq(2)'
          }
      })
  }).data(console.log);

[matthewmueller]

@rchipka thanks for chiming in! it definitely puts more emphasis on using a GraphiQL or some other GraphQL-aware IDE. Using one of those would improve the dev experience for crawling most websites.

Actually now wondering if we should just be providing primitives to do this stuff, rather than some fancy control flow libraries. Basically more building blocks that you can put together to handle more complicated login flows.

[rchipka]

@matthewmueller No problem. I was just checking out how things are going on x-ray and this issue stood out to me.

I think you should keep it as actual JS code and not a domain specific language in a string. You can implement a similar GraphQL look in pure JS by creating x.page(), x.query(), etc.