This is used to record the thoughts and results from the Drawabox exercise. At the same time, according to the 50% rule, I will be drawing figures or objects from the Atmosphere’s Sketch book as the random drawing. Maybe in the future, I will choose images from Pixiv.
Record of the house renting.
Notes about the drawabox.
- recommandations: https://drawabox.com/recommendations
- Dodge Charger
- Kia Soul
- Honda HR-V
- Subaru XV Crosstrek
- Honda Civic
- Toyota Corolla
- Mitsubishi Outlander
- Mazda CX-5
The Manager’s Path: A Guide for Tech Leaders Navigating Growth and Change is one of the books that I was referenced to prepare for the behavioral questions in the tech interviews. In general, there are four kinds of interview rounds: coding, system design, OOD, behavioral question. The BQ round is the only one associated with a manager, who has a lot of weight on the final decision to the offer.
Behavioral interview questions are those that focus on how you’ve handled various situations in the workplace and reveal your character traits, abilities and skills. These questions give an interviewer an idea of how you would behave if a similar situation were to arise, the logic being that your success in the past will show success in the future.
Unlike traditional interview questions, behavior job interview techniques look for concrete examples of skills and experiences that relate to the position. Your answers to these questions should provide a brief story that illustrates your skills and strengths as an employee. For each answer, give the interviewer the background to the story briefly, specific actions you took and the results.
This post includes the notes that I went through during the reading of the book as mentioned in title. I would like to paste one sentence that totally solved one of my questions regarding the differences between refactoring and design patterns:
When my coauthors and I wrote Design Patterns, we mentioned that design patterns provide targets for refactorings. However, identifying the target is only one part of the problem; transforming your code so that you get there is another challenge.
Dragonfly - an intelligent P2P based image and file distribution system.
Now the client of Dragonfly will random read and write disk multiple times during the downloading process.
For directly using
dfget to download a file:
dfgetrandom writes a piece into disk after downloading it
dfget serverrandom reads the piece from disk to share it
dfgetsequential reads the file from disk after downloading to do checksum
And for using
dfdaemon to pull images, there’re extra disk IO by Dragonfly:
dfdaemonsequential reads the file from disk to send it to
It’s not a problem when the host has a local disk. But it will be a potential
bottleneck when the Dragonfly client runs on a virtual machine with a cloud disk, all the disk IO will become network IO which has a bad performance when read/write at the same time.
So a solution is needed to reduce the IO times generated by Dragonfly.
As proposed in the Issue #1164, the P2P streaming is come up for the problem discussed above. And my work in ASoC 2020 is to implement part of the streaming download method. The task can be summaried into four aspects, which are supernode scheduler, dfget downloader, dfget uploader, IPC between user and Dragonfly stream dfget.
The work in supernode can be categorized into two respectives:
Maintain the sliding window in supernode.
- Initialize: The size of window will be registered in the dfget task registration phrase. And then, the window state is recorded in the ProgressManager as the SyncMap indexed by ClientID. The size of window is staic once it is recorded by supernode.
- Update: The dfget client will report to the supernode about the result of downloading for every piece. In the
reportAPI of supernode server, the window will be updated based on the received piece number and piece status. To be specific, the window keeps sliding until the unacknowledged piece. It should be noted that the supernode is using the sender window, which has the range of
[una, una + wnd).
- Finish: Update the
DeleteCIDmethod of ProgressManager to delete the sliding window state if it is in stream mode.
Schedule the pieces according to the window state
The only modification that I made in this part lies at the
GetPieceProgressByCIDmethod of ProgressManager. The available pieces in regular mode means the success pieces which are not running; while for the stream mode, the available pieces means the cached pieces which are not running. After the modification, the ProgressManager would only return the available pieces inside the window when stream mode is on.
A new kind of piece status is created: UNCACHED. When the piece is downloaded successfully, I assume that it will sotred into the cache immediately. Afterwards, the piece maybe popped out of the cache. And in this case, the handler of supernode server
supernode/server/router.go:101would be called to change the state of the piece.
It should be noted that, the scheduler in stream mode currently uses the same scheduler as the regular mode, which may demands future optimization.
Since the client stream writer has been implemented under the
p2p_downloader folder, which means that the pieces downloading process has been finished. My work here is to fullfill the task of handling the successfully downloaded pieces to uploader. Besides that, in the perparation phase at
dfget/core/core.go:171, the registration of stream task is necessary for uploader. Here are all the modifications:
doDownloadmethod is refactored. One new interface is defined as
DownloadTimeoutTask. The interface has two implementations, which are regular downloader and the stream downloader.
Start()method of the
StreamDownloadTimeoutTaskstruct will call the
RunStreammethod of the downloader, and then pass the stream reader to the
startWritermethod is used to fetch the pieces from the stream reader concurrently, and then upload the pieces to uploader in order. The logic of the
startWriteris referenced from the stream downloading method in CDN manager at
The unit test has been added.
The previous uploader is solely used for the regular downloading method. For the stream uploader, I have added new APIs for it. Apart from that, the cache manager is added to manage the cache pieces.
dfget/core/api/supernode_api.go:359: A new kind of updating piece status API has been implemented. It is used to change the peice status from SUCCESSFUL to UNCACHED.
dfget/core/api/uploader_api.go:108: The uploader API
RegisterStreamTaskis added to register the stream task at cache manager. The initialization of the according entry at cache manager is finshed after the call.
dfget/core/api/uploader_api.go:127: The uploader API
DeliverPieceToUploaderis used to handle the piece from downloader to uploader.
dfget/core/uploader/cache.go:60: The FIFO cache manager is created to control the piece caches. It would store and pop the piece in the style of FIFO. It should be noted that the cache manager is using the receiver window, which has the range of
The unit test has been added.
TODO: GC for cache manager
This part demands further discussion. Currently, I propose that the DFGET can directly output the content of the pieces to stdout, and then the user who calls the DFGET command, can redirect the stdout of the DFGET command to the write side of the pipe by generally supported method popen.
After that, the user can directly read from the pipe, and get the successfully downloaded content. Since the method uses the unnamed pipe, the whole process is not related with the file system.
|07/07 - 07/14||Community Bonding & Source Code Reading
|07/15 - 07/27||Research on the Scheduler of Supernode for Stream Downloading
Commit: Support Stream Mode in Supernode
|07/28 - 08/10||Discuss the Stream Implementation with a Mentor from Ant.
Commit: Implement Work Flow of Stream Mode in DFGET
|08/11 - 08/26||Research on the IPC between the User and Stream DFGET.
commit: Add Unit Test and Fix Bugs for Integration Test
Here is the changes of code lines after the commits: (+2033, -77).
The Overrall Proposal: https://github.com/dragonflyoss/Dragonfly/issues/1436#issue-659970568
Design of Supernode Scheduler: https://github.com/dragonflyoss/Dragonfly/pull/1447#issue-459701444
Design of DFGET downloader, uploader and IPC between user and DFGET: https://github.com/dragonflyoss/Dragonfly/pull/1447#issuecomment-681619384
Github Commits: https://github.com/dragonflyoss/Dragonfly/pull/1447/commits
The tests have been implemented into the test files. And the annotation inside the source files would be necessary for the developer to test.
The highlights of my work is that I have implemented almost all of the downloading in sream mode. The sum of the modification code lines are (2033 + 77) = 2110. And I believe that the stream mode will definitely improve the efficiency of the Dragonfly downloading process in the case of the remote file system.
What’s more, this will be one of the features that Dragonfly owns compared to the traditional P2P systems.
The whole ASoC experience benefits me a lot. First of all, I have done much research work during the application phase, which makes me familiar with the traditional distributed systems. And then, during the development phase, I learnt the system design, concurrent programming in golang, start-of-the-art architecture of Dragonfly and the group work experience with mentors in Alibaba.
Mentors are really good at what they are working for: by the simple discussion, they can always quickly find the weakness of my design and point out the correct direction of the next move.
In a word, it is so lucky for me to be chosen to be part of the 2020 ASoC program.
This post is going to record the note of design pattern of object-oriented programming. By reading the book, the following ability should be acquired:
- Finding Appropriate Objects: The hard part about OO design is de-composing a system into objects. Design patterns help you identify less-obvious abstractions and the objects that can capture them.
- Determining Object Granularity
- Specifying Object Interfaces:
The set of all signatures defined by an object’s operations is called the
interfaceto the object.
- Specifying Object Implementations
The reference material is listed below:
- the Github Respository: https://github.com/tmrts/go-patterns
- textbook: Design Patterns: Elements of Reuseable Object-Oriented Software
Christopher Alexander says, “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice”
Gamma, Erich. Design Patterns (Addison-Wesley Professional Computing Series) (p. 24). Pearson Education. Kindle Edition.
Research done for the design of Dragonfly Streaming Scheduler.
Here P2P Streaming means that the data downloaded by the peer which will then be passed to the target place, is not going to stored in the local file system. The data would be maintained in the memory, and then ditectly send to the target node.
This is necessary for the P2P downloading when the peer server have a cloud disk. In that case, the process of writing the data to the file system of peer server would be limited to the network I/O. But if the P2P streaming is adopted, the writing process would be eliminated.
In this post, I would do research on how to design the scheduler in the supernode/server to distribute the pieces among peer servers.