spronkey

the official docs don’t do a particularly good job of the mock object api

spronkey

does anyonek now if there are phpunit API docs available anywhere

nightjarnz

Pretty sure Heyday had a module for that too, although might be SS3 based and unmaintained.

rudiger

think I found what you’re referring to and looks like similar concept though I think it’s not injecting itself into SS so I’m back to my current problem

Firesphere

It's been done before

rudiger

that looks almost the same as what I was doing but it looks to work for 3, would have issues in 4

brett.tasker

What files are you having issues with?

Assets, Themes (Javascript / CSS), etc,

Depending on the system, you may need to take different approaches.

brett.tasker

E.g. for Asset (/assets), I would recommend using a single datasource mounted to your multi-server architecture.

brett.tasker

This way you don't have to worry about replication or differences between servers.

rudiger

yeah there is one mounted but it’s replication is about a second or 2, leading to the issue.

brett.tasker

@rudiger When you say replication, are you talking about delays in updates? Or are you syncing between 2 different file systems?

E.g. Do you A: (e.g. Network Attached Storage) Filesystem A -> Mount Server A Filesystem A -> Mount Server B

OR

Filesystem A -> Mount Server A Filesystem B -> Mount Server B

  • Replication Filesystem A <-> Filesystem B
rudiger

network file store and two servers in separate data centres

rudiger

when you load the CMS it combines the required files

brett.tasker

Where does it put those combined files? (E.g. what directory?)

rudiger

not sure the exact one atm but it lives in the assets folder

brett.tasker

Where is your Network storage mounted to?

(E.g. to assets/ or the whole codebase?)

brett.tasker

Your unfortunately not going to fix this issue with code (as the delay is going to cause problems that code won't fix).

So focus should be on reducing the delay in your infrastructure

rudiger

doing the hash rather than timestamp means that it’s unique per code (which only changes per deploy) rather than time.

brett.tasker

Only way I can think of doing it, is to have Silverstripe "regenerate" the Combined assets if the "Hashed" value of the content does not match the Hashed value in the request.

But this would add additional stress onto the request workflow (generating assets) for when syncing is too slow

brett.tasker

Do you not have any dynamic content in your "CombinedFiles" ? (E.g. Input fields in CMS that affect the content of these files)

rudiger

it’s only a problem after a code deploy where the code changes

brett.tasker

Then pre-warming your CombinedFiles is probably the best way to resolve this.

brett.tasker

You can use the Hash method as well (like with SS3), but this will add "regeneration overhead" to your request cycle.

By pre-warming, it will create the files during deployment, (rather than on-request) which means the 2second sync delay should not be an issue

rudiger

I tried that in SS3 without the hash function and it still caused issues

brett.tasker

That would only cause issues if the files were stored outside of your network storage (E.g. different on each server) OR they were requested before it had time to update on storage.

rudiger

looking at your code for multi-sever we did the same thing and it fixes it without performance issues

brett.tasker

There is one part confusing me. Under what condition would the timestamp change on the CombinedFiles?

E.g. Under what action would you get an issue?

Is it because when a user "Saves" changes to a CMS page, it forcefully re-generates these files (giving them a new timestamp)?

If so, you could potentially look at having the CMS Admin system only update these files if the content has changed (Comparing new file content with existing file content).

But unfortunately I am not as fluent in SS4 as SS3 currently (Working mostly on infrastructure now).

My recommendation would be to work on that 2second delay in your Network storage. Anything greater than 10ms is a pretty bad performance for NFS. As if you resolve this issue, then all your problems should be solved.

rudiger

from memory what happens is cms loads the html which requires js eg test.js?m=123

In that process it’s combining the file test.js, browser is now requesting test.js?m=123 from a different server, (this bit I’m sketchy on because it’s been 3 years) because this file doesn’t exist yet it returns html js include of test.js?m=124, it’s combining test.js and it just keeps bouncing from server to server

rudiger

even if the NFS is a few ms delay you run the risk of having it increment the second counter, just looking at that link you sent

rudiger

mmmm, I wonder if that fixes it anyway, guessing it’s a check for if the file exists before trying to create it

rudiger

guess I won’t know until I test it behind a LB

brett.tasker

The m= is not a counter.

In SS3 - It tracked specifically the "modified date" of the file (E.g. last time it was modified).

In SS4 - It appears to track the Hash of the file content (SHA1).. So only time this would change is if the content of the file changes.

Are you having issues in SS4? Or have you yet to test it?

The "multi-server" module I created for SS3 was to change the m= functionality to "Hash" (like SS4) rather than modified date.

rudiger

yeah the m in my example is just meant to represent a timestamp, couldn’t be bothered typing a correct one

👍 (1)
rudiger

SS4 has a lot of references to it hashing and the doco even talks about the config needed for a distributed environment but in the front end it still has ?m=timestamp

rudiger

but it’s possible it’s fixed it on the backend rather than doing an md5/sha1 on the frontend

rudiger

I’ll deploy it and see what happens, might revisit this if it continues to be a problem

brett.tasker

What you mean by "Front end it still has ?m=timestamp".

Are you sure its not ?m=Hashstring

rudiger

an example js path: resources/silverstripe/campaign-admin/client/dist/js/bundle.js?m=1584438524

brett.tasker

That's not a CombinedFile?

That is a composer install asset from "admin" module

brett.tasker

Is there any other Requirements_Backend in your codebase other than the default Framework one?

What framework version are you using?

rudiger

sorry back. Not atm, I’m trying to get the one from SS3 working but theres still issues.

brett.tasker

What happens if you remove the module completely?

rudiger

it’s not on for what I’m working on atm, the m=1584423458 is standard behaviour

rudiger

it’s not being called when I’m loading things that didn’t work in SS3. I’ll have to do more testing on a LB environment to see whats going on.

brett.tasker

Ahh I found out why.

using combine for files you will get a Hash result.

However using javascript() to define a resource it will use "modified time" as the cachebreaker.

https://github.com/silverstripe/silverstripe-framework/blob/1e01deea39f54ef69cadbad8fc6d8a211fcb5ba4/src/Control/SimpleResourceURLGenerator.php#L110

Show 1 attachment(s)
brett.tasker

So what you could do, is define your own ResourceURLGenerator (E.g. MyResourceURLGenerator.php ) that extends SimpleResourceURLGenerator,

Then override the public function urlForResource() and modify

  1. $query .= "m=" . filemtime($absolutePath);

to be

  1. $query .= "m=" . sha1_file($absolutePath);
brett.tasker

Then you just need to tell it to load your Generator.

  1. SilverStripe\Core\Injector\Injector:
  2. SilverStripe\Core\Manifest\ResourceURLGenerator:
  3. class: SilverStripe\Control\MyResourceURLGenerator
  4. properties:
  5. NonceStyle: mtime
brett.tasker

But that is a bit "Hacky".

In your MyResourceURLGenerator.php, you would probably want to replace setNonceStyle to allow for ''sha1_file' .

Then in the urlForResource() add in a new case statement for switch ($this->nonceStyle) to handle sha1_file

brett.tasker

Then you could set NonceStyle: sha1_file in your YML to make it cleaner

rudiger

thanks a heap for that, going to give that a try.

brett.tasker

👍 It would be even better if you create a PR back to framework to add this functionality in to the framework by default.

rudiger

oh ok? I’ve worked with other frameworks that use the hash rather than the timestamp due to time being something thats not reliable to identify if a file has changed.

rudiger

I know for passwords but we’re just looking for something reliably unique given x content

rudiger

there’s a higher chance of collision but the chances of 2 files having a collision given the large amount of content would be pretty low

brett.tasker

sha1 is the faster method for hashing compared to md5

brett.tasker

> MD5 is 7.6% slower than SHA-1 for short strings and 1.3% for longer strings.

brett.tasker

Through PHP they seem about the same speed. You could add 2x case options to your switch (1 for sha1_file and 1 for md5_file ) so that people could choose

rudiger

reading a bit more into it might be a bit more cpu architecture. think more modern ones have improved sha1 performance which makes sense.

brett.tasker

https://github.com/silverstripe/silverstripe-framework/pull/9509

Show 1 attachment(s)
brettt89

Add support for Sha1 and MD5 hashing when requesting a resource via ResourceURLGenerator.
E.g.
resource.js?m=&lt;sha1/md5/mtime&gt;

Hide attachment content
rudiger

oh nice, thanks for that. I had got it all working a couple of hours ago but better as a core feature


Show less replies
rudiger

hey all, when dealing with SS 3 on a distributed environment I had to write a plugin that basically created a hash of the file rather than using a timestamp because if a timestamp was used one server would say a slightly different time then the other which would cause an infinite loop of trying to load the assets. SS4 seems vastly different in the backend processing of those files so having some issues modifying it however in the docos is sort of mentioned with some config flags but they don’t do anything. Anyone have knowledge about this? Doco is at https://docs.silverstripe.org/en/4/developer_guides/templates/requirements/

Show 1 attachment(s)
docs.silverstripe.org  
Requirements

Developer and user documentation for the Silverstripe CMS and framework.

Hide attachment content