Migration from custom sources
To migrate files from custom sources, you need to create your own implementation of SourceStorageAdapter. The adapters prepare migration plans with the lists of categories, folders and assets to migrate. They also expose a method allowing the migrator to download files from the storage.
To create a new adapter, a directory must first be created in the adapters/ directory.
The next step is to create the Adapter.ts file. This is the entry point of the adapter and you cannot change this filename. To create the file, you may use the boilerplate below:
import {
ISourceStorageAdapter,
IMigrationPlan,
ISourceCategory,
ISourceFolder,
ISourceAsset
} from '@ckbox-migrator';
export default class ExampleAdapter implements ISourceStorageAdapter {
public readonly name: string = 'The name of your system';
public async loadConfig( plainConfig: Record<string, unknown> ): Promise<void> {
// Load and validate configuration.
}
public async verifyConnection(): Promise<void> {
// Verify if you can connect and authorize to your source storage.
}
public async prepareMigrationPlan(): Promise<IMigrationPlan> {
// Scan your source storage and identify the categories,
// folders and assets to migrate.
return [
categories: [],
assets: []
];
}
public async getAsset( downloadUrl: string ): Promise<IGetAssetResult> {
// Get an asset content from you source storage.
}
}
# Loading the configuration
To load the configuration, you need to implement the loadConfig( plainConfig: Record<string, unknown> ) method. It is the first step performed in the migration process. The parameter provided to the method is a source.options property from config.json file.
To validate the configuration, we recommend using the class-validator and class-transformer libraries. They are already bundled with the migrator so you do not need to install them separately.
First you need to create the configuration class. It will contain the config properties and validation rules.
class ExampleConfig {
@IsString()
@IsDefined()
public readonly foo: string;
}
Then, you can map the config using the plainToInstance and validate it using the validateOrReject method.
export default class ExampleAdapter implements ISourceStorageAdapter {
// ...
private _config: ExampleConfig;
public async loadConfig( plainConfig: Record<string, unknown> ): Promise<void> {
this._config = plainToInstance( CKFinderConfig, plainConfig );
await validateOrReject( this._config );
}
Now, you can configure your adapter. You need to set source.type to the name of the folder where your adapter is created, and set a configiguration compatible with ExampleConfig in source.options.
{
"source": {
"type": "example",
"options": {
"foo": "bar"
}
},
"ckbox": {
...
}
}
To build the application, use the following command:
npm run build:adapters
# Verify connection
Verify connection is a step used to check if the migrator can establish the connection to the source storage. You can make a request to any endpoint in your source system to check if authorization passes.
import fetch, { Response } from 'node-fetch';
export default class ExampleAdapter implements ISourceStorageAdapter {
// ...
const { serviceOrigin } = this._config;
const response: Response = await fetch( `${ serviceOrigin }/status` );
if ( !response.ok ) {
throw new Error( `Failed to connect to the Example service at ${ serviceOrigin }.` );
}
# Create the migration plan
Before the migration starts, the adapter should return the migration plan, the structure containing the list of categories, folders and assets to create.
You should request list of all resources from the source storage and map to the following format.
export interface IMigrationPlan {
readonly categories: ISourceCategory[];
readonly assets: ISourceAsset[];
}
The migration plan should be returned by the prepareMigrationPlan of the migrator:
import crypto from 'node:crypto';
export default class ExampleAdapter implements ISourceStorageAdapter {
// ...
public async prepareMigrationPlan(): Promise<IMigrationPlan> {
const categoryId: string = crypto.randomUUID();
return [
categories: [{
id: categoryId,
name: 'Example category',
allowedExtensions: ['txt'],
folders: []
}],
assets: [{
id: crypto.randomUUID(),
name: 'File',
extension: 'txt',
location: { categoryId }
downloadUrl: 'http://localhost/file.txt'
downloadUrlToReplace: 'http://localhost/file.txt'
}]
];
}
# Categories and folders
Each category should have unique identifier (it does not matter if it is generated by the source system or using a random ID generator in the adapter), the name, the list of allowed extensions (remember that it should cover at least extensions that will be uploaded to this category) and the folders tree.
export interface ISourceCategory {
readonly id: string;
readonly name: string;
readonly allowedExtensions: string[];
readonly folders: ISourceFolder[];
}
If your source system has a similar concept to categories (like “resource types” in CKFinder), you can map them directly to categories. If your system is much like a filesystem, you can treat categories as a top level folders.
To create the folders tree, you need to assign folders to the folders property in the category. Each folder should have its own identifier that is unique in the category (for example, a path to the folder or a randomly generated ID), the scope and the name. Folders can contain child folders.
export interface ISourceFolder
readonly id: string;
readonly name: string;
readonly childFolders: ISourceFolder[];
}
# Assets
Each category should have:
- Unique identifier - it may be a path to the file or randomly generated ID, same as with folders.
- File name.
- File extension.
- Location (id of the category and optionally id of the folder).
- URL that the migrator should use to download the file.
- URL that is used in your system and should be replaced with a new URL after migration. It may be the same as the URL used for download, but it does not need to. For example,
downloadUrlmay point to the API in the company network anddownloadUrlToReplaceto a publicly available endpoint.
export interface ISourceAsset {
readonly id: string;
readonly name: string;
readonly extension: string;
readonly location: ISourceLocation;
readonly downloadUrl: string;
readonly downloadUrlToReplace: string;
}
export interface ISourceLocation {
readonly categoryId: string;
readonly folderId?: string;
}
# Migrating assets
When the migration plan is ready, the migrator knows what categories, folders and assets should be created, but it does not know the files content yet. To provide the file content, you need to return the file stream from getAsset() method:
import fetch, { Response } from 'node-fetch';
export default class ExampleAdapter implements ISourceStorageAdapter {
// ...
public async getAsset( downloadUrl: string ): Promise<IGetAssetResult> {
const response: Response = await fetch( downloadUrl );
if ( !response.ok ) {
throw new Error(
`Failed to fetch file from the Example service at ${ downloadUrl }. ` +
`Status code: ${ response.status }. ${ await response.text() }`
);
}
return {
stream: response.body,
responsiveImages: []
}
}
# Responsive images
If your source system provided URLs for responsive images, it’s worth to return URLs of the responsive versions in the responsiveImages property of IGetAssetResult. Thanks to this, the migrator can add the links to the list of URLs mappings.
import fetch, { Response } from 'node-fetch';
export default class ExampleAdapter implements ISourceStorageAdapter {
// ...
public async getAsset( downloadUrl: string ): Promise<IGetAssetResult> {
const response: Response = await fetch( downloadUrl );
if ( !response.ok ) {
throw new Error(
`Failed to fetch file from the Example service at ${ downloadUrl }. ` +
`Status code: ${ response.status }. ${ await response.text() }`
);
}
return {
stream: response.body,
responsiveImages: [
{
width: 100,
url: `${ downloadUrl }/w_100`
},
{
width: 300,
url: `${ downloadUrl }/w_300`
}
// ...
]
}
}