nicolaas

How can I find a file in the Filesystem based on path

nicolaas

yeah - we will start with a problem definition. I am writing a task that can analyse assets pre-and-post upgrade to check things like: (a) is it in DB, (b) is it in file system (c) etc...

theruss

Well $file->exists() will do that for you (File::exists() also consults the F/S as well as the DB.

nicolaas

We will in due course - but we have two projects now where the files did not migrate very well at all. I was wondering if perhaps we are using the wrong version or missing something vital. Most of it runs smoothly, but some files are a mess.

nicolaas

Hi Everyone, we are having some issues with the file upgrader (https://github.com/silverstripe/silverstripe-assets/blob/1/src/Dev/Tasks/FileMigrationHelper.php et al.) not working seamless - is there anything out there that can help?

Show 1 attachment(s)
src/Dev/Tasks/FileMigrationHelper.php


<?php
namespace SilverStripe\Assets\Dev\Tasks;

use LogicException;
use Psr\Log\LoggerInterface;
use Psr\Log\NullLogger;
use SilverStripe\Assets\File;
use SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy;
use SilverStripe\Assets\FilenameParsing\FileResolutionStrategy;
use SilverStripe\Assets\FilenameParsing\LegacyFileIDHelper;
use SilverStripe\Assets\Flysystem\FlysystemAssetStore;
use SilverStripe\Assets\Folder;
use SilverStripe\Assets\Storage\AssetStore;
use SilverStripe\Assets\Storage\FileHashingService;
use SilverStripe\Core\Config\Configurable;
use SilverStripe\Core\Environment;
use SilverStripe\Core\Injector\Injectable;
use SilverStripe\Core\Injector\Injector;
use SilverStripe\ORM\DataList;
use SilverStripe\ORM\DataObject;
use SilverStripe\ORM\DataQuery;
use SilverStripe\ORM\DB;
use SilverStripe\ORM\ValidationException;
use SilverStripe\Versioned\Versioned;

/**
 * Service to help migrate File dataobjects to the new APL.
 *
 * This service does not alter these records in such a way that prevents downgrading back to 3.x
 */
class FileMigrationHelper
{
    use Injectable;
    use Configurable;

    private static $dependencies = [
        'logger' => '%$' . LoggerInterface::class . '.quiet',
    ];

    /**
     * @var LoggerInterface
     */
    private $logger;

    /**
     * @var FlysystemAssetStore
     */
    private $store;

    /**
     * If a file fails to validate during migration, delete it.
     * If set to false, the record will exist but will not be attached to any filesystem
     * item anymore.
     *
     * @config
     * @var bool
     */
    private static $delete_invalid_files = true;

    public function __construct()
    {
        $this->logger = new NullLogger();
    }

    /**
     * @param LoggerInterface $logger
     * @return $this
     */
    public function setLogger(LoggerInterface $logger)
    {
        $this->logger = $logger;

        return $this;
    }

    /**
     * Perform migration
     *
     * @param string $base Absolute base path (parent of assets folder). Will default to PUBLIC_PATH
     * @return int Number of files successfully migrated
     */
    public function run($base = null)
    {
        $this->store = Injector::inst()->get(AssetStore::class);
        if (!$this->store instanceof AssetStore || !method_exists($this->store, 'normalisePath')) {
            throw new LogicException(
                'FileMigrationHelper: Can not run if the default asset store does not have a `normalisePath` method.'
            );
        }

        if (empty($base)) {
            $base = PUBLIC_PATH;
        }

        // Set max time and memory limit
        Environment::increaseTimeLimitTo();
        Environment::setMemoryLimitMax(-1);
        Environment::increaseMemoryLimitTo(-1);

        /** @var FileHashingService $hasher */
        $hasher = Injector::inst()->get(FileHashingService::class);
        $hasher::flush();

        $this->logger->notice('Step 1/2: Migrate 3.x legacy files to new format');
        $ss3Count = $this->ss3Migration($base);

        $this->logger->notice('Step 2/2: Migrate files in <4.4 format to new format');

        $ss4Count = 0;
        if (class_exists(Versioned::class) && File::has_extension(Versioned::class)) {
            Versioned::prepopulate_versionnumber_cache(File::class, Versioned::LIVE);
            Versioned::prepopulate_versionnumber_cache(File::class, Versioned::DRAFT);

            $stages = [Versioned::LIVE => 'live', Versioned::DRAFT => 'draft'];
            foreach ($stages as $stageId => $stageName) {
                $this->logger->info(sprintf('Migrating files in the "%s" stage', $stageName));
                $count = Versioned::withVersionedMode(function () use ($stageId, $stageName) {
                    Versioned::set_stage($stageId);
                    return $this->normaliseAllFiles(sprintf('on the "%s" stage', $stageName));
                });
                if ($count) {
                    $this->logger->info(sprintf('Migrated %d files in the "%s" stage', $count, $stageName));
                } else {
                    $this->logger->info(sprintf(
                        'No files required migrating in the "%s" stage',
                        $stageName
                    ));
                }

                $ss4Count += $count;
            }
        } else {
            $ss4Count = $this->normaliseAllFiles('');
            $this->logger->info(sprintf('Migrated %d files', $ss4Count));
        }

        return $ss3Count + $ss4Count;
    }

    protected function ss3Migration($base)
    {
        // Check if the File dataobject has a "Filename" field.
        // If not, cannot migrate
        /** @skipUpgrade */
        if (!DB::get_schema()->hasField('File', 'Filename')) {
            return 0;
        }

        // Check if we have files to migrate
        $totalCount = $this->getFileQuery()->count();
        if (!$totalCount) {
            $this->logger->info('No files required migrating');
            return 0;
        }

        $this->logger->debug(sprintf('Migrating %d files', $totalCount));

        // Create a temporary SS3 Legacy File Resolution strategy for migrating SS3 Files
        $initialStrategy = $this->store->getPublicResolutionStrategy();
        $ss3Strategy = $this->buildSS3MigrationStrategy($initialStrategy);
        if (!$ss3Strategy) {
            $this->logger->warning(
                'Skipping this step because the asset store is using an unsupported public file ' .
                'resolution strategy.'
            );
            return 0;
        }
        $this->store->setPublicResolutionStrategy($ss3Strategy);

        // Force stage to draft
        if (class_exists(Versioned::class) && File::has_extension(Versioned::class)) {
            $originalState = Versioned::get_reading_mode();
            Versioned::set_stage(Versioned::DRAFT);
        }

        // Set up things before going into the loop
        $ss3Count = 0;
        $originalState = null;

        // Loop over the files to migrate
        try {
            foreach ($this->getLegacyFileQuery() as $file) {
                // Bypass the accessor and the filename from the column
                $filename = $file->getField('Filename');
                $success = $this->migrateFile($base, $file, $filename);
                if ($success) {
                    $ss3Count++;
                }
            }
        } finally {
            // Reset back to our initial state no matter what
            if (class_exists(Versioned::class)) {
                Versioned::set_reading_mode($originalState);
            }
            $this->store->setPublicResolutionStrategy($initialStrategy);
        }

        // Show summary of results
        if ($ss3Count > 0) {
            $this->logger->info(sprintf('%d 3.x legacy files have been migrated.', $ss3Count));
        } else {
            $this->logger->info(sprintf('No 3.x legacy files required migrating.', $ss3Count));
        }

        return $ss3Count;
    }

    /**
     * Construct a temporary SS3 File Resolution Strategy based off the provided initial strategy.
     * If `$initialStrategy` is not suitable for a migration, we return null.
     * We're overriding the helpers to avoid doing unnecessary checks.
     * @param FileResolutionStrategy $initialStrategy
     * @return int|FileIDHelperResolutionStrategy
     */
    private function buildSS3MigrationStrategy(FileResolutionStrategy $initialStrategy)
    {
        // If the project is using a custom FileResolutionStrategy, we can't be confident that our migration won't
        // break stuff, so let's bail
        if (!$initialStrategy instanceof FileIDHelperResolutionStrategy) {
            return null;
        }

        // Let's make sure the initial strategy …
Hide attachment content
nicolaas

@here - what the best place to ask an assets upgrader question?