Disable image caching #377

sjernigan · 2017-10-20T18:55:37Z

I've built an app that's main function is capture a frame from a web cam, associate it with some meta data, save and name the image based on an ID in the meta-data. Works great until I try to recapture an image. The image caching optimization prevents the new frame from being shown. I understand I could name the new picture something else but that is one of main jobs of this app. Is there a way to disable or clear the image caching?

ccoupe · 2017-10-20T19:14:14Z

It's not the first time that someone asked for this. Your example describes why this is needed.

ccoupe · 2017-10-20T19:26:36Z

You can clear the cache if you know the secret. Many parts of Shoes are in Ruby. The Maintain shoes screen (we called it the cobbler) has a button to clear cache and it written in ruby. deep inside Shoes.App/Contents/MacOs/lib/shoes/ is cobbler.rb and data.rb. image.rb is called from inside the the downloading code so you don't want to modify it but you may want to look at it.

sjernigan · 2017-10-20T20:36:11Z

Got it. Thanks

sjernigan · 2017-10-20T21:32:49Z

Well I thought I did. Something funky is going on. I was trying to confirm the key format but got caught up by my first issue. Images loaded from local disk do not show up in the cobbler 'select image to delete' list and don't seem to get deleted when I use delete all. However, something more broad might be going on. The Delete From Cache does not do what I would expect. Here's my test procedure: Load cobbler, click 'Splash' and also load the sample 'image' app, load the default dino image, click about in cobbler until the 'select image to delete' list refreshes and you see the dino URL, close the sample image app, DISCONNECT all internet access, Delete From Cache 'All' images (either with or without the 'Remove File? option), reopen the sample 'image' app, redownload the default dino image, WHAM - it shows back up despite no internet access. It does not show back up in the cobbler list. I'll keep looking through the code but any insight would be appreciated.

ccoupe · 2017-10-20T22:22:12Z

Keep in mind there is a internal copy as well as the cache files. If you have two processes, and one deletes from the db - the other process won't know that. It's even more complicated in the C code: An image appears to have 3 or 4 states of cache that I need to suss out.

If you are in a hurry, then you want to modify your app to delete the cached file (or the db) after you download an image. The 'db' is just sdbm gem. Before you load another image you need to insure the previous image is not in memory. How? Not Sure. Things are slightly different depending on whether the image was loaded from a file or downloaded from a url (you can two dino.jpg in the cache if they came from different sources. Speaking of that, how are you getting the images off the webcam? File reading or downloading?

ccoupe · 2017-10-21T02:46:01Z

I've got a test script. The db cache persists across Shoes invocations. There is also the strange setup in ~/.shoes/+cache that the db hash points to that needs deleting (the actual image data files) otherwise Shoes may find it in some situations and not do the download. This does not remove
the in memory cache, yet.

I'm proposing add a style arg image <url>, cache: false if you don't want the image cached

Shoes.app do
  require 'shoes/data'
  require 'shoes/image'
  stack do
    flow do
      @el = edit_line "#{DIR}/static/shoes-icon-walkabout.png"
      #@el = edit_line "https://shoes.mvmanila.com/public/images/dino.jpg"
      @cb = check; para "Don't cache"
      button "(Re)load" do
        @img.clear
        @img.append do
          if @cb.checked?
            image @el.text, cache: false
          else
            image @el.text
          end
        end
      end
      button "Show cache" do
        @cview.clear
        @cview.append do
          eb = edit_box width: 400
          DATABASE.each do |key, value|
            eb.append "#{key} -> #{value}"
          end
        end
      end
      button "clear caches" do
        DATABASE.each do |k, val|
          v = val.split('|')
          path = Shoes::image_cache_path v[1], File.extname(k)
          File.delete path if File.exist? path
        end        
        DATABASE.clear
      end
    end
    @img = flow {} 
    @cview = flow {}
  end
end
end

sjernigan · 2017-10-21T03:03:30Z

File reading... The ruby opencv gem was not working nicely so I shell out to python to capture the image, perform face detection, cropping, and scaling.

I confirm that your test script demonstrates the problem. I like addition of the cache: false args for 90% of the cases. But I suspect there is still a need for a image.clear_cache or clear_cache(url) for times when I didn't know up front that I would need to reload an image.

BTW: thank you for being so responsive and helpful

IanTrudel · 2017-10-21T12:55:43Z

BTW: thank you for being so responsive and helpful

We are always happy to have new people in here. We do our best to improve Shoes and your feedback is valuable to us.

I'm proposing add a style arg image <url>, cache: false if you don't want the image cached

Sounds good. Would such cache enable/disable also apply to download?

IanTrudel · 2017-10-21T12:58:46Z

I would also like to suggest cache as a Shoes.app hash parameter, and/or app.cache/app.cache=, to enable or disable globally.

ccoupe · 2017-10-21T19:30:39Z

The image cache is a very ancient part of Shoes and has tendrils in unexpected places (pattern for example). At the moment I'm going attempt to not do the cache lookup and just proceed to load/download the file which will get cached but who cares if we never look at the cache.

Would such cache enable/disable also apply to download?

I don't think the download method uses a cache - not appropriate for those 5gb iso files we played with a few months ago.

to enable or disable globally.

I did think about a global setting and it may be needed if my simple scheme doesn't work.

ccoupe · 2017-10-21T23:44:45Z

@sjernigan I've got a beta for osx that should do what you want when loading images from files. Downloads can still get confused and stuck in the external cache so the task is incomplete. If you launch from the command line (cshoes) you'll get some progress/debugging messages. I also updated the test script above if you need it.

* image <path/to/file.png>, cache: false * still have work to do with the external cache and downloads.

ccoupe · 2017-10-23T02:06:39Z

app.cache/app.cache=

Very likely to happen. There are many C functions and Ruby methods that need to know not to cache - just finding them all has take a lot of gdb head scratching. In image.c we can save/restore global status if the image <url>, cache: t/f is specified.

sjernigan · 2017-10-23T14:23:19Z

Many thanks, I'll test today. For my case, a global flag will work wonderfully.

sjernigan · 2017-10-23T20:02:00Z

I can confirm that the "cache: t/f" flag is working as described. Thanks for the quick turn-around

* using typhoenus would totally rewrite this rat maze. #307 * Did I mention this is ugly?

* update manual * bug377.rb is needs deeper thought * would app.image_cache be a better name? Do I care?

ccoupe · 2017-10-24T06:37:10Z

@sjernigan - A question you don't have to answer but I'm curious about your Shoes application

I shell out to python to capture the image, perform face detection, cropping, and scaling.

I happen to have a logitech usb camera, rarely used. Your Shoes app intrigues me. Is it proprietary? Can you share it? I've got a raspberry pi 3 with Shoes and your project just screams for a raspberry introduction for Shoes in MagPi , for example. Collaboratively. Contact me at [email protected] if you want to discuss this privately.

ccoupe · 2017-10-25T03:45:53Z

Just when I thought I had something useful, there is osx and nsurl.m - much rethinking required. Time to get rid of that goto in shoes/image.c and the hangover from curl?

* rename shoes_queue_download to shoes_native_download * refactor to get rid of the goto. It was was more annoying than duplicating code. Separate code paths for cache, no-cache. * weird tmp bug with osx for http no cache - more to do * undo a lot of the last commit * clear cache in bugs/bug377.rb requests a restart. Yes, you do want that. * move extern shoes_cache_setting to app.h

IanTrudel · 2017-10-27T23:54:16Z

I happen to have a logitech usb camera, rarely used. Your Shoes app intrigues me. Is it proprietary? Can you share it?

Additionally, we could add some screenshots for Shoes website or something. I suggested in the past we have some kind of display of existing apps written in Shoes such as @dredknight wheel, Numinoes, etc. Perhaps something involving face detection and stuff would be cool as well. 😄

ccoupe · 2017-10-28T02:39:22Z

shoesrb.com is used by both shoes 3 and shoes 4 and needs some love from someone. I'm sure they wouldn't mind another page or two at the site for promotion, screenshots, links to to shoes apps. commercial or oss. Both projects need some promotion help - completely out of my wheelhouse.

dredknight · 2017-10-29T20:39:49Z

Btw I also think that Shoes needs some kind of page with screenshots of complete projects, examples and tutorials where people can also comment below and have some conversions between each other.
I am not a FE developer but I believe there are web site/forumish like templates that can be used without much sweat.

There is a similar story going with our mod. At first we were just a few people. And although some people claimed this is very neat work we got very little fans. Not that we cared as we did it for ourselves.
At some point we decided to go live on facebook and Moddb and this is when things got big very fast.
Our last release got 8000+ downloads, FB has 800+ fans where the FB activity is 10%+ of all fans which in my opinion I consider very high. It also brought in a lot of contributors which is the best part actually :).

Newbie devs just start and try to play with something. They do not have the mentality "Hey I can get to a forum and chat with people about that thing so I can improve my skills". They are just like 2-year olds playing with a wooden cubes. The only real user counter of shoes users is the download rate of the shoes.exe. How much dls has it accumulated for the last month? year?

Btw I do not mind maintaining Shoes FB page. It will give an easy point of access for many people plus it is some kind of advertisement.

Of course I can be wrong for all that I said above but it does not hurt giving a try.

ccoupe · 2017-10-30T01:48:14Z

Re: Image non-caching. I think I've got this working. I thinking of adding a app.clear_cache method that shoes/ruby can call to remove everything in the in memory cache (shoes_world->image_cache) and possibly the secondary sdbm cache. Most users wouldn't need it since they can turn the cache off and on as they like, but for bug377.rb and cobbler.rb it would help.

* add app.cache_clear to Shoes. Modify bug377.rb and cobbler.rb to use it. * add Manual description for cache_clear

ccoupe · 2017-10-30T06:11:12Z

Re: image non-caching. Betas available here As always, these have not been tested everywhere for everything. Yes, there is a freebsd.install. I'll merge this 'cache' branch to master based on feedback so test away! Please.

ccoupe · 2017-10-30T06:40:08Z

I am not a FE developer but I believe there are web site/forumish like templates that can be used without much sweat.

Forums are easy enough to install in a Wordpress blog. Pain in the ass (PITA) to configure and maintain. Not something I'm going to do again. However,

Btw I do not mind maintaining Shoes FB page. It will give an easy point of access for many people plus it is some kind of advertisement.

I like this idea! I'm not Facebook savy - I just argue politics with 'family' I don't know very well. :-) That said, I am a member of a group which doesn't need a lot of moderation. This could work. Several of my favorite restaurants have 'sites' with comments and pictures and menu images. I don't know if those are free or not any more - I grabbed TheCecil when they gave them away many years ago.

ccoupe · 2017-10-30T07:51:08Z

The only real user counter of shoes users is the download rate of the shoes.exe. How much dls has it accumulated for the last month? year?

About 4 or 5 users per day click the packaging 'select architecture' button - now that its fixed again. I get 50+ downloads a day for some version of Shoes 3. One half to two thirds of them are bots and a few would be downloads to complete the packaging. A lot of noise from China, Russia, Ukraine. I'd guess 10 real people a day download Shoes 3 but that's only a guess.

dredknight · 2017-10-30T12:26:56Z

Good :). I will prepare something and will show it to you. I will make your FB account a second admin for the purpose of posting.

IanTrudel · 2017-10-30T16:07:43Z

I'm not a big fan of dividing up the already fragmented community but a Facebook page might be a good thing. For example, members could post their own screenshots and videos to share. @dredknight you can also add me as admin.

A lot of noise from China, Russia, Ukraine.

Can you provide stats on this? Might be worthwhile to investigate. Maybe we could have the manual translated or something. Chinese and Russian people are very good contributors, very skillful as well.

ccoupe · 2017-10-30T21:55:39Z

I classify downloads awkwardly - too many edge cases and noise.From 2016-Feb-02 to yesterday

hacker 129
shoes  1968
maybe 11298
iffy  1789

Those are unique IP's. 'shoes' means they used the packager (or wrote their crawler in Ruby) 'Maybe' means it could be a user download or it could be a webcrawler. I often see one IP download windows, osx and a linux or two in one day. Legitimate? Then there are the people who download 3.2.25 and the browsers that chunk the download in two or more transactions. 'iffy' people broke some rule that make me pretty sure they are not legitimate. It's all in a 10MB sqlite3 db if someones wants to play with it - 2.1MB tgz

ccoupe · 2017-10-31T05:21:27Z

I'm not a big fan of dividing up the already fragmented community but a Facebook page might be a good thing.

I checked shoesrb.com to make sure they didn't setup a FB group long ago because it would be embarrassing to not know and duplicate/split the audience. They haven't and there are several places in the shoesrb.com Jekyll generated pages to refer to a new Facebook group without compromising integrity or look and feel of shoesrb.com. I do like that site design, someone did a good job. IMO. Not a fan of Jekyll but credit is due.

@dredknight, once you get the FB group setup, let me know and I or you can announce it on the mailing list and I can update the shoesrb pages (tastefully).

ccoupe · 2017-10-31T06:06:46Z

I'm not a big fan of dividing up the already fragmented community but a Facebook page might be a good thing.

I checked shoesrb.com to make sure they didn't setup a FB group long ago because it would be embarrassing to not know and duplicate/split the audience. They haven't and there are several places in the shoesrb.com Jekyll generated pages to refer to a new Facebook group without compromising integrity or look and feel of shoesrb.com. I do like that site design, someone did a good job. IMO. Not a fan of Jekyll but credit is due.

@dredknight, once you get the FB group setup, let me know and I or you can announce it on the mailing list and I can update the shoesrb pages (tastefully).

IanTrudel · 2017-10-31T09:53:50Z

Sounds very good. I also recommend to come up with a good name for the group url. The one that is gonna end up as https://www.facebook.com/<group name>. Interestingly, https://www.facebook.com/Shoes is available. Hurry up. haha

dredknight · 2017-10-31T10:23:01Z

Sure this is what I was thinking about as well :D.

dredknight · 2017-10-31T10:29:14Z

Unfortunately shoes was not available, nor shoes3 so I made it like this https://www.facebook.com/rubyshoes3/

IanTrudel · 2017-10-31T10:32:40Z

It's just so so... but at least descriptive.

I joined the page.

dredknight · 2017-10-31T10:34:19Z

Well I just made it so besides the avatar there is nothing. I need some time to find the correct visuals and see what else can be put in a more representative manner.

IanTrudel · 2017-10-31T10:40:44Z

No worries! Take your time! Your help is always appreciated. :)

ccoupe · 2017-11-01T06:07:48Z

@dredknight , your invitation for me to be an admin got this when I clicked on the FB email.

The invitation to become an admin of Shoes has expired. Please ask a current admin of the Page to add you.

I'm remembering why I don't use FB very much.

dredknight · 2017-11-01T17:29:11Z

@ccoupe @backorder admin invitations sent once again!

IanTrudel · 2017-11-01T17:50:19Z

@dredknight thanks, I added a link to https://shoesrb.com and liked the page Numinoes (made with Shoes). You can repost photos/videos from Numinoes on Shoes if you'd like and your own project. Perhaps you could create an issue here specific for Facebook where we can talk about it? We are totally off topic in here, e.g. disable image chache.

dredknight · 2017-11-01T19:01:19Z

@backorder, here you go -> #380
I am not even halfway with the FB page,I have some weird issue with creativity that it takes time. Sometimes a lot of ideas come in a sec while other times I sit and observe the thing for ages until something vague starts to shape up.

You can check how a developed fb page looks like. It is not a big deal but it took a while. -> https://www.facebook.com/mightandmagicheroes5.5/?ref=bookmarks

sjernigan · 2017-11-02T21:02:15Z

shoes-3.3.4-osx-10.9.tgz tested, caching clearing working. Thanks again

ccoupe self-assigned this Oct 20, 2017

ccoupe added the Normal label Oct 20, 2017

ccoupe added this to the 3.3.4 milestone Oct 22, 2017

ccoupe referenced this issue Oct 22, 2017

skip cache whem loading images from files #379

f3ec387

* image <path/to/file.png>, cache: false * still have work to do with the external cache and downloads.

ccoupe pushed a commit that referenced this issue Oct 24, 2017

for #377, #323 cache and no-cache are working, file and http

df5331c

* using typhoenus would totally rewrite this rat maze. #307 * Did I mention this is ugly?

ccoupe referenced this issue Oct 24, 2017

add app.cache/app.cache

996d23f

* update manual * bug377.rb is needs deeper thought * would app.image_cache be a better name? Do I care?

ccoupe pushed a commit that referenced this issue Oct 30, 2017

issue #377, #323 - finishing up?

0751630

* add app.cache_clear to Shoes. Modify bug377.rb and cobbler.rb to use it. * add Manual description for cache_clear

ccoupe mentioned this issue Oct 30, 2017

Rename this repository to 'shoes' shoes/shoes4#713

Closed

ccoupe closed this as completed Nov 2, 2017

Disable image caching #377

Disable image caching #377

Comments

sjernigan commented Oct 20, 2017 • edited Loading

ccoupe commented Oct 20, 2017

ccoupe commented Oct 20, 2017 • edited Loading

sjernigan commented Oct 20, 2017

sjernigan commented Oct 20, 2017

ccoupe commented Oct 20, 2017

ccoupe commented Oct 21, 2017 • edited Loading

sjernigan commented Oct 21, 2017

IanTrudel commented Oct 21, 2017

IanTrudel commented Oct 21, 2017

ccoupe commented Oct 21, 2017 • edited Loading

ccoupe commented Oct 21, 2017 • edited Loading

ccoupe commented Oct 23, 2017

sjernigan commented Oct 23, 2017

sjernigan commented Oct 23, 2017

ccoupe commented Oct 24, 2017

ccoupe commented Oct 25, 2017 • edited Loading

IanTrudel commented Oct 27, 2017

ccoupe commented Oct 28, 2017

dredknight commented Oct 29, 2017 • edited Loading

ccoupe commented Oct 30, 2017 • edited Loading

ccoupe commented Oct 30, 2017

ccoupe commented Oct 30, 2017 • edited Loading

ccoupe commented Oct 30, 2017 • edited Loading

dredknight commented Oct 30, 2017

IanTrudel commented Oct 30, 2017

ccoupe commented Oct 30, 2017 • edited Loading

ccoupe commented Oct 31, 2017

ccoupe commented Oct 31, 2017

IanTrudel commented Oct 31, 2017

dredknight commented Oct 31, 2017

dredknight commented Oct 31, 2017

IanTrudel commented Oct 31, 2017

dredknight commented Oct 31, 2017

IanTrudel commented Oct 31, 2017

ccoupe commented Nov 1, 2017

dredknight commented Nov 1, 2017

IanTrudel commented Nov 1, 2017

dredknight commented Nov 1, 2017 • edited Loading

sjernigan commented Nov 2, 2017

sjernigan commented Oct 20, 2017 •

edited

Loading

ccoupe commented Oct 20, 2017 •

edited

Loading

ccoupe commented Oct 21, 2017 •

edited

Loading

ccoupe commented Oct 21, 2017 •

edited

Loading

ccoupe commented Oct 21, 2017 •

edited

Loading

ccoupe commented Oct 25, 2017 •

edited

Loading

dredknight commented Oct 29, 2017 •

edited

Loading

ccoupe commented Oct 30, 2017 •

edited

Loading

ccoupe commented Oct 30, 2017 •

edited

Loading

ccoupe commented Oct 30, 2017 •

edited

Loading

ccoupe commented Oct 30, 2017 •

edited

Loading

dredknight commented Nov 1, 2017 •

edited

Loading