JS practice: learn about 5 file upload scenarios in one article (React + Koa Implementation)
preface
Today, I'd like to share with you five common file upload scenarios, from simple to deep, and take them carefully (the sample code warehouse also comes with a server implemented by koa, which is responsible for saving the code. This article will briefly introduce the implementation of the front-end part)
text
The front-end code shown in this article uses React as the carrier, but the ability of file upload itself has nothing to do with React
1. Single file upload
The single file is very simple. I believe everyone knows something
- /fe/src/tests/Single.tsx
The main core logic is to use the < input type = "file" > tag to select and upload files
import React, { ChangeEvent, useState } from 'react'
import { group } from '../utils/msg'
interface UploadProps {
url: string
body: FormData
}
export const uploadREQ = ({ url, body }: UploadProps) => {
return fetch(url, {
method: 'POST',
body,
})
}
const Single = () => {
const [filePath, setFilePath] = useState('')
const onInputFileChange = (e: ChangeEvent<HTMLInputElement>) => {
console.log('selected file:', e.target.files[0])
}
const upload = () => {
const input = document.createElement('input')
input.type = 'file'
input.addEventListener('change', e => {
const file = (e.target as HTMLInputElement).files[0]
console.log('selected file:', file)
const formData = new FormData()
formData.append('file', file, `1_single_${file.name}`)
const url = 'http://localhost:3001/upload/single'
uploadREQ({ url, body: formData }).then(async res => {
const result = await res.json()
group(`[response] ${url}`, () => {
console.log(result)
})
setFilePath(result.url)
})
})
input.click()
}
return (
<div>
<h1>File upload - 1: Single file upload</h1>
<input
id="input-file"
type="file"
onChange={onInputFileChange}
/>
<input
id="input-file2"
type="file"
accept=".png,.jpg"
onChange={onInputFileChange}
/>
<button id="btn-file3" onClick={upload}>
Click to upload
</button>
<h4>
file path:{' '}
<a target="_blank" href={filePath}>
{filePath}
</a>
</h4>
</div>
)
}
export default Single
Leaving aside some unimportant code, we can see that the following sentences are the core code
- e.target.fils or inputElement.files to get the selected file
const file = (e.target as HTMLInputElement).files[0]
- Wrap the file in a FormData and pass it to the server as the body of the post request
const formData = new FormData()
formData.append('file', file, `1_single_${file.name}`)
uploadREQ({ url, body: formData })
2. Multi file upload
- /fe/src/tests/Multiple.tsx
In fact, multiple files are also very simple. You can select multiple files by adding a multiple attribute to < input type = "file" multiple >, which is essentially the same as a single file
import React, { ChangeEvent } from 'react'
import { group } from '../utils/msg'
import { uploadREQ } from './Single'
interface UploadFilesProps {
url: string
files: File[]
prefix: string
fromDir?: boolean
}
export const uploadFiles = ({
url,
files,
prefix,
fromDir = false,
}: UploadFilesProps) => {
const formData = new FormData()
files.forEach(file => {
const fileName = fromDir
? // @ts-ignore
file.webkitRelativePath.replace(/\//g, `@${prefix}_`)
: `${prefix}_${file.name}`
formData.append('files', file, fileName)
})
console.log('upload files:', formData.getAll('files'))
return uploadREQ({ url, body: formData })
}
const Multiple = () => {
const onFileChange = (e: ChangeEvent<HTMLInputElement>) => {
const files = Array.from(e.target.files)
const url = 'http://localhost:3001/upload/multiple'
uploadFiles({
url,
files,
prefix: '2_multiple',
}).then(async res => {
const result = await res.json()
group(`[response] ${url}`, () => {
console.log(result)
})
})
}
return (
<div>
<h1>File upload - 2: Multi file upload</h1>
<input
id="input-files"
type="file"
multiple
onChange={onFileChange}
/>
</div>
)
}
export default Multiple
3. Upload multiple files by directory
- /fe/src/tests/Directory.tsx
Uploading by directory, especially the WebKit directory attribute, is generally supported by most browsers, although it is non-standard
import React, { ChangeEvent } from 'react'
import { group } from '../utils/msg'
import { uploadFiles } from './Multiple'
const Directory = () => {
const onFileChange = (e: ChangeEvent<HTMLInputElement>) => {
const files = Array.from(e.target.files)
const url = 'http://localhost:3001/upload/multiple'
uploadFiles({
url,
files,
prefix: '3_directory',
fromDir: true,
}).then(async res => {
const result = await res.json()
group(`[response] ${url}`, () => {
console.log(result)
})
})
}
return (
<div>
<h1>File upload - 3: Upload by directory</h1>
<input
id="input-files"
type="file"
// @ts-ignore
webkitdirectory="true"
onChange={onFileChange}
/>
</div>
)
}
export default Directory
4. Multi file composite compressed package upload
- /fe/src/tests/Zip.tsx
Fourth, we inherit the previous multi file selection. Whether we upload multiple files or directories, we also use another jszip package to generate a compressed package, and then the compressed package can be uploaded as a single file
The first is the compression method
const ZIP = (
zipName: string,
files: File[],
options: JSZip.JSZipGeneratorOptions = {
type: 'blob',
compression: 'DEFLATE',
}
): Promise<Blob> => {
return new Promise((resolve, reject) => {
const zip = new JSZip()
files.forEach(file => {
const path = (file as any).webkitRelativePath
zip.file(path, file)
})
zip.generateAsync(options).then((bolb: Blob) => {
resolve(bolb)
})
})
}
The following is the core code of the page
const Zip = () => {
const onFileChange = async (e: ChangeEvent<HTMLInputElement>) => {
const files = Array.from(e.target.files)
// @ts-ignore
const dirName = files[0].webkitRelativePath.split('/')[0]
const zipName = `${dirName}.zip`
const zipFile = await ZIP(zipName, files)
const formData = new FormData()
formData.append('file', zipFile, zipName)
const url = 'http://localhost:3001/upload/single'
uploadREQ({
url,
body: formData,
}).then(async res => {
const result = await res.json()
group(`[response] ${url}`, () => {
console.log(result)
})
})
}
return (
<div>
<h1>File upload - 4: Compressed file upload</h1>
<input
id="input-files"
type="file"
// @ts-ignore
webkitdirectory="true"
onChange={onFileChange}
/>
</div>
)
}
export default Zip
5. Upload large files in blocks
- /fe/src/tests/BigFile.tsx
The last part is a little more complicated. We explain it paragraph by paragraph
There are several main ideas to realize large file upload
- Generate the file characteristic value (MD5) and query the back-end for the existence of the file
- If it does not exist, the file will be chunked and uploaded one by one (multiple blocks can be uploaded concurrently)
- Finally, the files are combined at the back end to generate the original file
The front end is still relatively simple. Firstly, since the browser will actually limit the number of http concurrent requests in the same domain, we can implement a concurrent request management system ourselves
// Concurrent request pool
const asyncPool = async (
poolLimit: number,
tasks: any[],
iteratorFn: (task: any, tasks?: any[]) => Promise<any>
) => {
const waiting = [];
const executing = [];
for (const task of tasks) {
// Create asynchronous task
const p = Promise.resolve().then(() => iteratorFn(task, tasks));
waiting.push(p);
// The number of tasks exceeds the pool size
if (poolLimit <= tasks.length) {
const e = p.then(() =>
executing.splice(executing.indexOf(e), 1)
);
executing.push(e);
if (executing.length >= poolLimit) {
await Promise.race(executing);
}
}
}
return Promise.all(waiting);
};
The second tool method is to generate eigenvalues according to the file content. spark-md5 is used here
import SparkMD5 from 'spark-md5';
// Calculation file md5
const calcFileMD5 = (file: File): Promise<string> => {
return new Promise((resolve, reject) => {
const chunks = getChunks(file);
let currentChunk = 0;
const spark = new SparkMD5.ArrayBuffer();
const fileReader = new FileReader();
fileReader.onload = (e) => {
spark.append(e.target.result as ArrayBuffer);
currentChunk++;
if (currentChunk < chunks) {
loadNext();
} else {
resolve(spark.end());
}
};
fileReader.onerror = (e) => {
reject(fileReader.error);
fileReader.abort();
};
function loadNext() {
const start = currentChunk * chunkSize,
end = Math.min(file.size, start + chunkSize);
fileReader.readAsArrayBuffer(file.slice(start, end));
}
loadNext();
});
};
Next, enter the main business process. The first method is to request the backend to check whether the file exists
interface ICheckFileExistRes {
code: number;
data: {
isExists: boolean;
[key: string]: any;
};
}
// Check whether the file exists
const checkFileExist = (
name: string,
md5: string,
chunks: number
): Promise<ICheckFileExistRes> => {
const params = qs.stringify({
n: name,
m: md5,
c: chunks,
});
const url = `http://localhost:3001/upload/checkExist?${params}`;
return fetch(url)
.then((res) => res.json())
.then((res) => {
group(`[response] ${url}`, () => {
console.log(res);
});
return res;
});
};
The second method is to upload files in batches when they do not exist
const chunkSize = 1024 * 1024; // 1MB
const getChunks = (file: File) => {
return Math.ceil(file.size / chunkSize);
};
interface IUploadChunkProps {
url: string;
chunk: any;
chunkId: number;
chunks: number;
fileName: string;
fileMD5: string;
}
/**
* Upload file block
* @param param0
* @returns
*/
const uploadChunk = ({
url,
chunk,
chunkId,
chunks,
fileName,
fileMD5,
}: IUploadChunkProps) => {
const formData = new FormData();
formData.set('file', chunk, `${fileMD5}-${chunkId}`);
formData.set('chunks', chunks + '');
formData.set('name', fileName);
formData.set('timestamp', Date.now().toString());
return fetch(url, {
method: 'POST',
body: formData,
}).then((res) => res.json());
};
interface IUploadFileProps {
file: File;
fileMD5: string;
chunkIds: string[];
chunkSize?: number;
poolLimit?: number;
}
/**
* Large file upload
*/
const uploadFile = ({
file,
fileMD5,
chunkIds,
chunkSize = 1 * 1024 * 1024, // 1MB
poolLimit = 3,
}: IUploadFileProps) => {
const chunks = getChunks(file);
return asyncPool(
poolLimit,
// @ts-ignore
[...new Array(chunks).keys()],
(i: number) => {
if (chunkIds.includes(i + '')) {
return Promise.resolve();
}
const start = i * chunkSize;
const end = i + 1 === chunks ? file.size : start + chunkSize;
const chunk = file.slice(start, end);
return uploadChunk({
url: 'http://localhost:3001/upload/chunk',
chunk,
chunkId: i,
chunks,
fileName: file.name,
fileMD5,
});
}
);
};
The last part is the main component code
const BigFile = () => {
const inputRef = useRef<HTMLInputElement>();
const upload = async () => {
// Get basic file information
const file = inputRef.current.files[0];
const fileMD5 = await calcFileMD5(file);
console.log('select file:', file);
console.log('fileMD5:', fileMD5);
// Check whether the file exists
const res = await checkFileExist(
file.name,
fileMD5,
getChunks(file)
);
console.log('res', res);
// Re upload file
if (res.code && res.data.isExists) {
console.log(`file exist: ${res.data.url}`);
} else {
const result = await uploadFile({
file,
fileMD5,
chunkIds: res.data.chunkIds as string[],
});
console.log('result', result);
}
};
const clear = () => {
inputRef.current.value = '';
};
return (
<div>
<h1>File upload - 5: Large file upload</h1>
<input id="input-files" type="file" ref={inputRef} />
<button onClick={upload}>Upload</button>
<button onClick={clear}>Clear</button>
</div>
);
};
export default BigFile;
epilogue
In fact, there is no difficult part in the concept. Interested students can go to the code warehouse to see the implementation, or pull down and run by themselves
Other resources
Reference connection
Complete code example
https://github.com/superfreeeee/Blog-code/tree/main/front_end/javascript/js_upload