Implementation principle of golang coroutine

Core idea

To understand the implementation of coroutines, you first need to understand three very important concepts in go, they are G, M and P,
Those who haven't seen the golang source code may be unfamiliar with them. These three items are the most important components of the coroutine, and they are ubiquitous in the golang source code.

G (goroutine)

G is the initial text of goroutine, goroutine can be interpreted as a managed lightweight thread, goroutine is created using the go keyword.

For example, func main() { go other() }, this code creates two goroutine s,
One is main, the other is other, note that main itself is also a goroutine.

The creation, sleep, resume, and stop of goroutine s are all managed by the go runtime.
When a goroutine performs an asynchronous operation, it will enter a sleep state and resume after the operation is completed, without occupying system threads.
When a goroutine is newly created or resumed, it will be added to the run queue, waiting for M to be taken out and run.

M (machine)

M is the header text of machine, which is equivalent to the system thread in the current version of golang.
M can run two kinds of codes:

  • go code, that is, goroutine, M requires a P to run go code

  • Native code, such as blocking syscalls, M does not need P to run native code

M will take out G from the run queue, and then run G, if G finishes running or goes to sleep, it will take out the next G run from the run queue, and the cycle goes on and on.
Sometimes G needs to call some native code that cannot avoid blocking. At this time, M will release the P it holds and enter the blocking state, and other M will obtain this P and continue to run the G in the queue.
go needs to ensure that there are enough M to run G, not to let the CPU idle, and to ensure that the number of M is not too large.

P (process)

P is the header text of the process, representing the resources that M needs to run G.
Some articles explaining coroutines understand P as the cpu core, which is actually wrong.
Although the number of P is equal to the number of cpu cores by default, it can be modified through the environment variable GOMAXPROC. In actual operation, P has nothing to do with cpu cores.

P can also be understood as a mechanism to control the parallelism of go code,
If the number of P is equal to 1, it means that there can only be at most one thread (M) to execute the go code,
If the number of P is equal to 2, it means that currently there can only be at most two threads (M) executing the go code.
The number of threads executing native code is not controlled by P.

Because only one thread (M) can own P at the same time, the data in P is lock free, and the efficiency of reading and writing these data will be very high.

data structure

Before explaining the workflow of coroutines, you also need to understand some internal data structures.

State of G

  • Idle (_Gidle): Indicates that G has just been newly created and has not yet been initialized

  • To be run (_Grunnable): Indicates that G is in the running queue, waiting for M to be taken out and run

  • Running (_Grunning): Indicates that M is running this G, at this time M will have a P

  • In system call (_Gsyscall): Indicates that M is running the system call initiated by G, at this time M does not own P

  • Waiting (_Gwaiting): Indicates that G is waiting for some conditions to complete, at this time G is not running nor in the running queue (may be in the waiting queue of the channel)

  • Aborted (_Gdead): Indicates that G is not used and may have been executed (and is waiting for the next reuse in freelist)

  • Stack copying (_Gcopystack): Indicates that G is acquiring a new stack space and copying the original content (to prevent GC scanning)

State of M

M does not have the same state flags as G and P, but an M can be considered to have the following states:

  • Spinning: M is getting G from the run queue, at this time M will have a P

  • Executing go code: M is executing go code, then M will have a P

  • In executing native code: M is executing native code or blocking syscall, at this time M does not own P

  • In hibernation: M will enter hibernation when it finds that there is no G to be run, and add it to the free M linked list. At this time, M does not own P

The state of spinning is very important, whether it needs to wake up or create a new M depends on the number of M in the current spin.

State of P

  • Idle (_Pidle): When M finds that there is no G to run, it will go to sleep. At this time, the P owned by M will become idle and added to the idle P list

  • Running (_Prunning): When M has a P, the state of this P will become running, and M running G will use the resources in this P

  • In system call (_Psyscall): When go calls native code, and the native code in turn calls go code, the P used will change to this state

  • GC stop (_Pgcstop): When gc stops the whole world (STW), P will change to this state

  • Aborted (_Pdead): When the number of P changes during runtime, and the excess P will become this state when the number decreases

local run queue

There are multiple run queues in go that can save G to be run (_Grunnable), which are the local run queue and global run queue in each P respectively.
When the G to be run is enqueued, it will be added to the local running queue of the current P first. When M obtains the G to be run, it will also be preferentially obtained from the local running queue of the owned P.
Enqueuing and dequeuing the local run queue does not require thread locks.

The number of local run queues is limited, and when the number reaches 256, it will be queued to the global run queue.
The data structure of the local run queue is a circular queue, consisting of a 256-length array and two sequence numbers (head, tail).

When M obtains G from P's local running queue, if it finds that the local queue is empty, it will try to steal half of G from other P's.
This mechanism is called Work Stealing, as detailed in the code analysis below.

global run queue

The global run queue is stored in the global variable sched, and thread locks are required to enqueue and dequeue the global run queue.
The data structure of the global run queue is a linked list, which consists of two pointers (head, tail).

Free M linked list

When M finds that there is no G to be run, it will go to sleep and add it to the free M linked list, which is stored in the global variable sched.
The sleeping M will wait for a semaphore (m.park), and the sleeping M will use this semaphore.

go needs to ensure that there are enough M to run G, which is achieved through this mechanism:

  • After enqueuing the G to be run, if there is currently no spinning M but there is an idle P, wake up or create a new M

  • When M leaves the spin state and is ready to run the dequeued G, if there is currently no spin M but there is an idle P, wake up or create a new M

  • When M leaves the spin state and is ready to sleep, it will check all run queues again after leaving the spin state, and re-enter the spin state if there is a G to be run

Since "enqueue G to run" and "M leave spin state" will happen at the same time, go will use this check order:

Enqueue G to run => memory barrier => check the number of M currently spinning => wake up or create a new M
Decrease number of M for current spins => memory barrier => check all runqueues for G to run => sleep

In this way, it can be ensured that there will be no G to be run into the queue, and there are idle resources P, but no M to execute.

Free P linked list

When all Gs in P's local run queue have finished running, and G cannot be obtained from other places,
M that owns P will release P and enter the sleep state, the released P will become idle and added to the idle P linked list, and the idle P linked list is stored in the global variable sched
The next time the G to be running joins the queue, if it finds that there is an idle P, but there is no M in the spinning, it will wake up or create a new M, M will have this P, and P will become the running state again.

Workflow (Overview)

The following figure is the possible working state of the coroutine. There are 4 Ps in the figure, among which M1~M3 are running G and will continue to obtain G from the running queue of the owned P after running:

Just looking at this picture may be a little hard to imagine the actual workflow, here I will explain it again based on the actual code:

package main

import (
    "fmt"
    "time"
)

func printNumber(from, to int, c chan int) {
    for x := from; x <= to; x++ {
        fmt.Printf("%d\n", x)
        time.Sleep(1 * time.Millisecond)
    }
    c <- 0
}

func main() {
    c := make(chan int, 3)
    go printNumber(1, 3, c)
    go printNumber(4, 6, c)
    _ = <- c
    _ = <- c
}

When the program starts, a G will be created first, pointing to main (actually runtime.main instead of main.main, explained later):
The dotted line in the figure refers to the address where G is to run or start running, not the current running address.

M will take this G and run:

At this point main will create a new channel and start two new G s:

Next G: main will get data from the channel, because it can't get it, G will save the state and become waiting (_Gwaiting) and add it to the channel's queue:

Because G:main saves the running state, it will continue to run from _ = <- c the next time it runs.
Next M will get G: printNumber from the run queue and run:

printNumber will print the number, write data to the channel after completion,
When writing data, it is found that there is a waiting G in the channel, and the data will be handed over to this G, which will make G to be run (_Grunnable) and put it back into the running queue:

Next, M will run the next G: printNumber, because a buffer of size 3 is specified when the channel is created, and data can be directly written to the buffer without waiting:

Then printNumber runs, and only G: main is left in the run queue:

Finally, M takes out G: main and runs it, and it will continue to run from the last interrupted position _ <- c:

The result of the first _ <- c has been set before, this statement will execute successfully.
The second _ <- c will find that there is a buffered 0 in the channel when it is acquired, so the result is this 0, no need to wait.
Finally, the main execution is completed, and the program ends.

One might wonder what would happen if a _ <- c was added at the end, then since all G s are in a wait state, go will detect and report a deadlock:

fatal error: all goroutines are asleep - deadlock!

Start code analysis

This is the end of the explanation of the concept. From here, the implementation code in go will be analyzed. We need to understand some basic content first.

assembly code

From the following go code:

package main

import (
    "fmt"
    "time"
)

func printNumber(from, to int, c chan int) {
    for x := from; x <= to; x++ {
        fmt.Printf("%d\n", x)
        time.Sleep(1 * time.Millisecond)
    }
    c <- 0
}

func main() {
    c := make(chan int, 3)
    go printNumber(1, 3, c)
    go printNumber(4, 6, c)
    _, _ = <- c, <- c
}

The following assembly code can be generated (the platform is linux x64, using the default options, i.e. enable optimization and inlining):

(lldb) di -n main.main
hello`main.main:
hello[0x401190] <+0>:   movq   %fs:-0x8, %rcx
hello[0x401199] <+9>:   cmpq   0x10(%rcx), %rsp
hello[0x40119d] <+13>:  jbe    0x401291                  ; <+257> at hello.go:16
hello[0x4011a3] <+19>:  subq   $0x40, %rsp
hello[0x4011a7] <+23>:  leaq   0xb3632(%rip), %rbx       ; runtime.rodata + 38880
hello[0x4011ae] <+30>:  movq   %rbx, (%rsp)
hello[0x4011b2] <+34>:  movq   $0x3, 0x8(%rsp)
hello[0x4011bb] <+43>:  callq  0x4035a0                  ; runtime.makechan at chan.go:49
hello[0x4011c0] <+48>:  movq   0x10(%rsp), %rax
hello[0x4011c5] <+53>:  movq   $0x1, 0x10(%rsp)
hello[0x4011ce] <+62>:  movq   $0x3, 0x18(%rsp)
hello[0x4011d7] <+71>:  movq   %rax, 0x38(%rsp)
hello[0x4011dc] <+76>:  movq   %rax, 0x20(%rsp)
hello[0x4011e1] <+81>:  movl   $0x18, (%rsp)
hello[0x4011e8] <+88>:  leaq   0x129c29(%rip), %rax      ; main.printNumber.f
hello[0x4011ef] <+95>:  movq   %rax, 0x8(%rsp)
hello[0x4011f4] <+100>: callq  0x430cd0                  ; runtime.newproc at proc.go:2657
hello[0x4011f9] <+105>: movq   $0x4, 0x10(%rsp)
hello[0x401202] <+114>: movq   $0x6, 0x18(%rsp)
hello[0x40120b] <+123>: movq   0x38(%rsp), %rbx
hello[0x401210] <+128>: movq   %rbx, 0x20(%rsp)
hello[0x401215] <+133>: movl   $0x18, (%rsp)
hello[0x40121c] <+140>: leaq   0x129bf5(%rip), %rax      ; main.printNumber.f
hello[0x401223] <+147>: movq   %rax, 0x8(%rsp)
hello[0x401228] <+152>: callq  0x430cd0                  ; runtime.newproc at proc.go:2657
hello[0x40122d] <+157>: movq   $0x0, 0x30(%rsp)
hello[0x401236] <+166>: leaq   0xb35a3(%rip), %rbx       ; runtime.rodata + 38880
hello[0x40123d] <+173>: movq   %rbx, (%rsp)
hello[0x401241] <+177>: movq   0x38(%rsp), %rbx
hello[0x401246] <+182>: movq   %rbx, 0x8(%rsp)
hello[0x40124b] <+187>: leaq   0x30(%rsp), %rbx
hello[0x401250] <+192>: movq   %rbx, 0x10(%rsp)
hello[0x401255] <+197>: callq  0x4043c0                  ; runtime.chanrecv1 at chan.go:354
hello[0x40125a] <+202>: movq   $0x0, 0x28(%rsp)
hello[0x401263] <+211>: leaq   0xb3576(%rip), %rbx       ; runtime.rodata + 38880
hello[0x40126a] <+218>: movq   %rbx, (%rsp)
hello[0x40126e] <+222>: movq   0x38(%rsp), %rbx
hello[0x401273] <+227>: movq   %rbx, 0x8(%rsp)
hello[0x401278] <+232>: leaq   0x28(%rsp), %rbx
hello[0x40127d] <+237>: movq   %rbx, 0x10(%rsp)
hello[0x401282] <+242>: callq  0x4043c0                  ; runtime.chanrecv1 at chan.go:354
hello[0x401287] <+247>: movq   0x28(%rsp), %rbx
hello[0x40128c] <+252>: addq   $0x40, %rsp
hello[0x401290] <+256>: retq
hello[0x401291] <+257>: callq  0x4538d0                  ; runtime.morestack_noctxt at asm_amd64.s:365
hello[0x401296] <+262>: jmp    0x401190                  ; <+0> at hello.go:16
hello[0x40129b] <+267>: int3
hello[0x40129c] <+268>: int3
hello[0x40129d] <+269>: int3
hello[0x40129e] <+270>: int3
hello[0x40129f] <+271>: int3

(lldb) di -n main.printNumber
hello`main.printNumber:
hello[0x401000] <+0>:   movq   %fs:-0x8, %rcx
hello[0x401009] <+9>:   leaq   -0x8(%rsp), %rax
hello[0x40100e] <+14>:  cmpq   0x10(%rcx), %rax
hello[0x401012] <+18>:  jbe    0x401185                  ; <+389> at hello.go:8
hello[0x401018] <+24>:  subq   $0x88, %rsp
hello[0x40101f] <+31>:  xorps  %xmm0, %xmm0
hello[0x401022] <+34>:  movups %xmm0, 0x60(%rsp)
hello[0x401027] <+39>:  movq   0x90(%rsp), %rax
hello[0x40102f] <+47>:  movq   0x98(%rsp), %rbp
hello[0x401037] <+55>:  cmpq   %rbp, %rax
hello[0x40103a] <+58>:  jg     0x40112f                  ; <+303> at hello.go:13
hello[0x401040] <+64>:  movq   %rax, 0x40(%rsp)
hello[0x401045] <+69>:  movq   %rax, 0x48(%rsp)
hello[0x40104a] <+74>:  xorl   %ebx, %ebx
hello[0x40104c] <+76>:  movq   %rbx, 0x60(%rsp)
hello[0x401051] <+81>:  movq   %rbx, 0x68(%rsp)
hello[0x401056] <+86>:  leaq   0x60(%rsp), %rbx
hello[0x40105b] <+91>:  cmpq   $0x0, %rbx
hello[0x40105f] <+95>:  je     0x40117e                  ; <+382> at hello.go:10
hello[0x401065] <+101>: movq   $0x1, 0x78(%rsp)
hello[0x40106e] <+110>: movq   $0x1, 0x80(%rsp)
hello[0x40107a] <+122>: movq   %rbx, 0x70(%rsp)
hello[0x40107f] <+127>: leaq   0xb73fa(%rip), %rbx       ; runtime.rodata + 54400
hello[0x401086] <+134>: movq   %rbx, (%rsp)
hello[0x40108a] <+138>: leaq   0x48(%rsp), %rbx
hello[0x40108f] <+143>: movq   %rbx, 0x8(%rsp)
hello[0x401094] <+148>: movq   $0x0, 0x10(%rsp)
hello[0x40109d] <+157>: callq  0x40bb90                  ; runtime.convT2E at iface.go:128
hello[0x4010a2] <+162>: movq   0x18(%rsp), %rcx
hello[0x4010a7] <+167>: movq   0x20(%rsp), %rax
hello[0x4010ac] <+172>: movq   0x70(%rsp), %rbx
hello[0x4010b1] <+177>: movq   %rcx, 0x50(%rsp)
hello[0x4010b6] <+182>: movq   %rcx, (%rbx)
hello[0x4010b9] <+185>: movq   %rax, 0x58(%rsp)
hello[0x4010be] <+190>: cmpb   $0x0, 0x19ea1b(%rip)      ; time.initdone.
hello[0x4010c5] <+197>: jne    0x401167                  ; <+359> at hello.go:10
hello[0x4010cb] <+203>: movq   %rax, 0x8(%rbx)
hello[0x4010cf] <+207>: leaq   0xfb152(%rip), %rbx       ; go.string.* + 560
hello[0x4010d6] <+214>: movq   %rbx, (%rsp)
hello[0x4010da] <+218>: movq   $0x3, 0x8(%rsp)
hello[0x4010e3] <+227>: movq   0x70(%rsp), %rbx
hello[0x4010e8] <+232>: movq   %rbx, 0x10(%rsp)
hello[0x4010ed] <+237>: movq   0x78(%rsp), %rbx
hello[0x4010f2] <+242>: movq   %rbx, 0x18(%rsp)
hello[0x4010f7] <+247>: movq   0x80(%rsp), %rbx
hello[0x4010ff] <+255>: movq   %rbx, 0x20(%rsp)
hello[0x401104] <+260>: callq  0x45ad70                  ; fmt.Printf at print.go:196
hello[0x401109] <+265>: movq   $0xf4240, (%rsp)          ; imm = 0xF4240
hello[0x401111] <+273>: callq  0x442a50                  ; time.Sleep at time.go:48
hello[0x401116] <+278>: movq   0x40(%rsp), %rax
hello[0x40111b] <+283>: incq   %rax
hello[0x40111e] <+286>: movq   0x98(%rsp), %rbp
hello[0x401126] <+294>: cmpq   %rbp, %rax
hello[0x401129] <+297>: jle    0x401040                  ; <+64> at hello.go:10
hello[0x40112f] <+303>: movq   $0x0, 0x48(%rsp)
hello[0x401138] <+312>: leaq   0xb36a1(%rip), %rbx       ; runtime.rodata + 38880
hello[0x40113f] <+319>: movq   %rbx, (%rsp)
hello[0x401143] <+323>: movq   0xa0(%rsp), %rbx
hello[0x40114b] <+331>: movq   %rbx, 0x8(%rsp)
hello[0x401150] <+336>: leaq   0x48(%rsp), %rbx
hello[0x401155] <+341>: movq   %rbx, 0x10(%rsp)
hello[0x40115a] <+346>: callq  0x403870                  ; runtime.chansend1 at chan.go:99
hello[0x40115f] <+351>: addq   $0x88, %rsp
hello[0x401166] <+358>: retq
hello[0x401167] <+359>: leaq   0x8(%rbx), %r8
hello[0x40116b] <+363>: movq   %r8, (%rsp)
hello[0x40116f] <+367>: movq   %rax, 0x8(%rsp)
hello[0x401174] <+372>: callq  0x40f090                  ; runtime.writebarrierptr at mbarrier.go:129
hello[0x401179] <+377>: jmp    0x4010cf                  ; <+207> at hello.go:10
hello[0x40117e] <+382>: movl   %eax, (%rbx)
hello[0x401180] <+384>: jmp    0x401065                  ; <+101> at hello.go:10
hello[0x401185] <+389>: callq  0x4538d0                  ; runtime.morestack_noctxt at asm_amd64.s:365
hello[0x40118a] <+394>: jmp    0x401000                  ; <+0> at hello.go:8
hello[0x40118f] <+399>: int3

It doesn't matter if you can't understand these assembly codes now, and I will take a part from here to explain.

call specification

Different platforms have different calling conventions for functions.
For example, 32-bit parameters are passed through the stack, and return values ​​are passed through the eax register.
64-bit windows pass the first 4 parameters through rcx, rdx, r8, r9, pass the 5th parameter through the stack, and pass the return value through the eax register.
64-bit linux, unix pass the first 6 parameters through rdi, rsi, rdx, rcx, r8, r9, pass the 7th parameter through the stack, and pass the return value through the eax register.
Go does not use these calling conventions (unless it involves interacting with native code), go has its own set of calling conventions.

The calling specification of go is very simple. All parameters are passed through the stack, and the return value is also passed through the stack.
For example a function like this:

type MyStruct struct { X int; P *int }
func someFunc(x int, s MyStruct) (int, MyStruct) { ... }

The contents of the stack when the function is called are as follows:

It can be seen that the parameters and return values ​​are arranged from low to high. This is why the go function can have multiple return values. Because the return values ​​are all passed through the stack.
It should be noted that the "return address" here is on x86 and x64, the return address of arm will be saved through the LR register, and the content will be slightly different from here.
Also note that unlike c, the entire contents of the constructor will be copied to the stack when the constructor is passed, and performance will be affected if the constructor is large.

TLS

The full name of TLS is Thread-local storage, which represents local data in each thread.
For example, errno in standard c is a typical TLS variable, each thread has a separate errno, and writing it will not interfere with the value in other threads.
go relies heavily on the TLS mechanism when implementing coroutines, and will be used to obtain the current G in the system thread and the instance of M to which G belongs.

Because go does not use glibc, operating TLS will use the system's native interface, taking linux x64 as an example,
When go creates a new M, it will call the arch_prctl syscall to set the value of the FS register to the address of M.tls,
The FS register of each M in operation will point to the tls of their corresponding M instance. When the linux kernel schedules a thread, the FS register will switch with the thread.
In this way, the go code only needs to access the FS register to access thread-local data.

in the assembly code above

hello[0x401000] <+0>:   movq   %fs:-0x8, %rcx

will move the pointer to the current G from TLS to the rcx register.

stack expansion

Because coroutines in go are stackful coroutine s, each goroutine needs to have its own stack space,
The content of the stack space needs to be retained when the goroutine sleeps, and will be restored after the sleep is completed (the entire call tree is complete at this time).
This leads to a problem. There may be many goroutines at the same time. If each goroutine allocates enough stack space in advance, then go will use too much memory.

To avoid this problem, go only allocates a small stack space for goroutine s at the beginning, and its size is 2K in the current version.
When the function finds that the stack space is insufficient, it will apply for a new stack space and copy the original stack content.

in the assembly code above

hello[0x401000] <+0>:   movq   %fs:-0x8, %rcx
hello[0x401009] <+9>:   leaq   -0x8(%rsp), %rax
hello[0x40100e] <+14>:  cmpq   0x10(%rcx), %rax
hello[0x401012] <+18>:  jbe    0x401185                  ; <+389> at hello.go:8

It will check whether the comparison rsp is smaller than g.stackguard0 after subtracting a certain value. If it is less than or equal to, you need to call the morestack_noctxt function below.
Careful people may find that the compared value is inconsistent with the actual subtracted value. This is because a small amount of space will be reserved under stackguard0, and the comparison can be omitted if the reserved space is not exceeded at compile time.

Write Barrier

Because go supports parallel GC, GC scanning and go code can run at the same time. The problem is that the go code may change the object's dependency tree during the GC scanning process.
For example, when starting to scan, it is found that root objects A and B, B owns the pointer of C, GC scans A first, then B gives the pointer of C to A, GC scans B again, then C will not be scanned.
To avoid this problem, go enables Write Barrier during the marking phase of the GC.

After enabling the write barrier (Write Barrier), when B hands the pointer of C to A, the GC will think that the pointer of C is alive in this round of scanning,
Even though A may throw away C later, then C is recycled in the next round.
The write barrier is only enabled for pointers, and only during the marking phase of the GC, which usually writes the value directly to the target address:

Details about write barriers will be analyzed in the next article (GC).
It is worth mentioning that CoreCLR's GC also has a write barrier mechanism, which is the same as explained here.

Closure

The concept of closure itself should not need to be explained, let's actually take a look at how go implements closures:

package main

import (
    "fmt"
)

func executeFn(fn func() int) int {
    return fn();
}

func main() {
    a := 1
    b := 2
    c := executeFn(func() int {
        a += b
        return a
    })
    fmt.Printf("%d %d %d\n", a, b, c)
}

The output of this code is 3 2 3, which should not surprise anyone familiar with go.
The assembly code for the main function to execute the executeFn function is as follows:

hello[0x4a096f] <+47>:  movq   $0x1, 0x40(%rsp)          ; variable a equal to 1
hello[0x4a0978] <+56>:  leaq   0x151(%rip), %rax         ; register rax Equal to anonymous function main.main.func1 the address of
hello[0x4a097f] <+63>:  movq   %rax, 0x60(%rsp)          ; variable rsp+0x60 is equal to the address of the anonymous function
hello[0x4a0984] <+68>:  leaq   0x40(%rsp), %rax          ; register rax equal to variable a the address of
hello[0x4a0989] <+73>:  movq   %rax, 0x68(%rsp)          ; variable rsp+0x68 equal to variable a the address of
hello[0x4a098e] <+78>:  movq   $0x2, 0x70(%rsp)          ; variable rsp+0x70 equal to 2(variable b the value of)
hello[0x4a0997] <+87>:  leaq   0x60(%rsp), %rax          ; register rax equals address rsp+0x60
hello[0x4a099c] <+92>:  movq   %rax, (%rsp)              ; The first parameter is equal to the address rsp+0x60
hello[0x4a09a0] <+96>:  callq  0x4a08f0                  ; implement main.executeFn
hello[0x4a09a5] <+101>: movq   0x8(%rsp), %rax           ; register rax equal to return value

We can see that what is passed to executeFn is a pointer, and the content pointed to by the pointer is [the address of the anonymous function, the address of the variable a, the value of the variable b].
The reason why the variable a passes the address is that the modification of a in the anonymous function needs to be reflected on the original a.
The assembly code of the executeFn function to execute the closure is as follows:

hello[0x4a08ff] <+15>: subq   $0x10, %rsp                ; Allocate 0 on the stack x10 Space
hello[0x4a0903] <+19>: movq   %rbp, 0x8(%rsp)            ; put the original register rbp move to variable rsp+0x8
hello[0x4a0908] <+24>: leaq   0x8(%rsp), %rbp            ; put the variable rsp+0x8 move the address to the register rbp
hello[0x4a090d] <+29>: movq   0x18(%rsp), %rdx           ; put the first parameter(Closure)move the pointer to the register rdx
hello[0x4a0912] <+34>: movq   (%rdx), %rax               ; Move the pointer of the function in the closure to a register rax
hello[0x4a0915] <+37>: callq  *%rax                      ; Call the function in the closure
hello[0x4a0917] <+39>: movq   (%rsp), %rax               ; move return value to register rax
hello[0x4a091b] <+43>: movq   %rax, 0x20(%rsp)           ; put the register rax move to return value(after the parameter)
hello[0x4a0920] <+48>: movq   0x8(%rsp), %rbp            ; put the variable rsp+0x8 The value of the restore register rbp(restore the original rbp)
hello[0x4a0925] <+53>: addq   $0x10, %rsp                ; free stack space
hello[0x4a0929] <+57>: retq                              ; return from function

It can be seen that when the closure is called, the parameters are not passed through the stack, but through the register rdx. The assembly code of the closure is as follows:

hello[0x455660] <+0>:  movq   0x8(%rdx), %rax            ; Move the first argument to a register rax(variable a pointer)
hello[0x455664] <+4>:  movq   (%rax), %rcx               ; put the register rax Move the pointed value to the register rcx(variable a the value of)
hello[0x455667] <+7>:  addq   0x10(%rdx), %rcx           ; Add the second parameter to the register rcx(variable a the value of+variable b the value of)
hello[0x45566b] <+11>: movq   %rcx, (%rax)               ; put the register rcx move to register rax the value pointed to(The result of the addition is saved back to the variable a)
hello[0x45566e] <+14>: movq   %rcx, 0x8(%rsp)            ; put the register rcx Move to return result
hello[0x455673] <+19>: retq                              ; return from function

The transitivity of closures can be summarized as follows:

  • The content of the closure is [the address of the anonymous function, the parameters passed to the anonymous function (variable length)...]

  • Passing a closure to other functions passes a pointer to the "contents of the closure"

  • When the closure is called, a pointer to the "contents of the closure" is placed in the register rdx (in go this pointer is called the "context")

  • The closure will take the parameter from the register rdx

  • If the closure modifies the variable, the parameter in the closure will be a pointer instead of a value, and it will be modified to the original position when modified

Closure + goroutine

If you are careful, you may find that in the above example, the content of the closure is on the stack. What if instead of calling executeFn directly, go executeFn?
Changing the above code to go executeFn(func() ...) can generate the following assembly code:

hello[0x455611] <+33>:  leaq   0xb4a8(%rip), %rax        ; register rax equals type information
hello[0x455618] <+40>:  movq   %rax, (%rsp)              ; The first parameter is equal to the type information
hello[0x45561c] <+44>:  callq  0x40d910                  ; transfer runtime.newobject
hello[0x455621] <+49>:  movq   0x8(%rsp), %rax           ; register rax equal to return value(here called new object a)
hello[0x455626] <+54>:  movq   %rax, 0x28(%rsp)          ; variable rsp+0x28 equals new object a
hello[0x45562b] <+59>:  movq   $0x1, (%rax)              ; new object a is equal to 1
hello[0x455632] <+66>:  leaq   0x136e7(%rip), %rcx       ; register rcx equals type information
hello[0x455639] <+73>:  movq   %rcx, (%rsp)              ; The first parameter is equal to the type information
hello[0x45563d] <+77>:  callq  0x40d910                  ; transfer runtime.newobject
hello[0x455642] <+82>:  movq   0x8(%rsp), %rax           ; register rax equal to return value(here called new object fn)
hello[0x455647] <+87>:  leaq   0x82(%rip), %rcx          ; register rcx Equal to anonymous function main.main.func1 the address of
hello[0x45564e] <+94>:  movq   %rcx, (%rax)              ; new object fn+0 value equal to main.main.func1 the address of
hello[0x455651] <+97>:  testb  (%rax), %al               ; make sure the new object fn not equal to nil
hello[0x455653] <+99>:  movl   0x78397(%rip), %ecx       ; register ecx Equal to whether the write barrier is currently enabled
hello[0x455659] <+105>: leaq   0x8(%rax), %rdx           ; register rdx equals new object fn+0x8 the address of
hello[0x45565d] <+109>: testl  %ecx, %ecx                ; Determine whether the write barrier is currently enabled
hello[0x45565f] <+111>: jne    0x455699                  ; The logic behind is called when the write barrier is enabled
hello[0x455661] <+113>: movq   0x28(%rsp), %rcx          ; register rcx equals new object a
hello[0x455666] <+118>: movq   %rcx, 0x8(%rax)           ; set new object fn+0x8 The value of is equal to the new object a
hello[0x45566a] <+122>: movq   $0x2, 0x10(%rax)          ; set new object fn+0x10 is equal to 2(variable b the value of)
hello[0x455672] <+130>: movq   %rax, 0x10(%rsp)          ; The third parameter is equal to the new object fn(Extra parameters)
hello[0x455677] <+135>: movl   $0x10, (%rsp)             ; The first parameter is equal to 0 x10(function+the size of the parameter)
hello[0x45567e] <+142>: leaq   0x22fb3(%rip), %rax       ; The second parameter is equal to the address of a constant constructor
hello[0x455685] <+149>: movq   %rax, 0x8(%rsp)           ; The type of this construct is funcval, value is executeFn the address of
hello[0x45568a] <+154>: callq  0x42e690                  ; transfer runtime.newproc create new goroutine

We can see that the situation of goroutine + closure is more complicated. First, go will calculate the variable a and the closure will escape to the outside through escape analysis.
At this time, go will allocate the variable a and the closure on the heap. The two newobject calls above are the allocation of the variable a and the closure respectively.
When creating a goroutine, first pass in the size of the function + parameter (the above is 8+8=16), and then pass in the function + parameter, the above parameter is the address of the closure.

m0 and g0

There are also special M and G in go, which are m0 and g0.

m0 is the main thread after starting the program. The instance corresponding to this m will be in the global variable m0 and does not need to be allocated on the heap.
m0 is responsible for performing initialization operations and starting the first g, after which m0 is the same as any other m.

g0 is only responsible for scheduling G, g0 does not point to any executable function, each m will have its own g0,
The stack space of g0 is used when scheduling or system calls, and the g0 of the global variable is the g0 of m0.

If you understand the above content, you can start looking at the source code of golang.

program initialization

The entry point of the go program is runtime.rt0_go, and the process is:

  • Allocating stack space requires 2 local variables + 2 function parameters, and then aligns to 8

  • Save the incoming argc and argv to the stack

  • Update the value of stackguard in g0, stackguard is used to detect whether the stack space is insufficient, and new stack space needs to be allocated

  • Get the information of the current cpu and save it to each global variable

  • call _cgo_init if the function exists

  • Initialize the TLS of the current thread, set the FS register to m0.tls+8(-8 when acquired)

  • Test if TLS is working

  • Set g0 to TLS, indicating that the current g is g0

  • set m0.g0 = g0

  • set g0.m = m0

  • Call runtime.check to do some checks

  • Call runtime.args to save the incoming argc and argv to global variables

  • Call runtime.osinit to perform different initializations according to the system

    • Here (linux x64) the global variable ncpu is set equal to the number of cpu cores

  • Call runtime.schedinit to perform common initialization

    • There is a lot of processing here, which will initialize the stack space allocator, GC, generate P according to the number of cpu cores or the value of GOMAXPROCS, etc.

    • The processing of generating P is in procresize

  • Call runtime.newproc to create a new goroutine, pointing to runtime.main

    • The runtime.newproc function is also used when creating ordinary goroutine s, which will be explained in detail in the following "go Implementation"

  • Call runtime·mstart to start m0

    • After startup, m0 will continuously obtain G from the running queue and run, and will not return after runtime.mstart is called

    • The runtime.mstart function is the entry point of m (not just m0), which will be explained in detail in the "implementation of the scheduler" below

The first scheduled G will run runtime.main, the process is:

  • Mark the main function has been called, set mainStarted = true

  • Start a new M to execute the sysmon function, which monitors the global state and preempts the G that runs too long

  • Requires G must be executed on the current M (system main thread)

  • Call the runtime_init function

  • call the gcenable function

  • call the main.init function, if the function exists

  • G is no longer required to run on current M

  • If the program was compiled as a class library for c, return here

  • call main.main function

  • If a panic currently occurs, wait for the panic to be processed

  • Call exit(0) to exit the program

Definition of GMP

The definition of G is here.
The definition of M is here.
The definition of P is here.

The more important members of G are as follows

  • stack: the stack space currently used by g, which has two members: lo and hi

  • stackguard0: Check whether the stack space is enough value, below this value will expand the stack, 0 is used by go code

  • stackguard1: Check whether the stack space is enough value, below this value will expand the stack, 1 is used by native code

  • m: m corresponding to the current g

  • sched: The scheduling data of g, when g is interrupted, the current values ​​of pc and rsp will be saved here, and the values ​​here will be used when resuming operation

  • atomicstatus: the current state of g

  • schedlink: the next g, which will be used when g is in the linked list structure

  • preempt: whether g is being preempted

  • lockedm: Does g require to return to this M for execution, and sometimes g interrupts the recovery and requires the use of the original M to execute

The more important members of M are as follows

  • g0: special g used for scheduling, switch to this g when scheduling and executing system calls

  • curg: currently running g

  • p: currently owned P

  • nextp: When M wakes up, M will have this P

  • park: The semaphore used when M sleeps, and it will wake up when M wakes up

  • schedlink: the next m, which will be used when m is in the linked list structure

  • mcache: the local allocator used when allocating memory, the same as p.mcache (it will be copied when you have P)

  • lockedg: the corresponding value of lockedm

The more important members of P are as follows

  • status: the current status of p

  • link: the next p, which will be used when p is in the linked list structure

  • m: M who owns this P

  • mcache: the local allocator used when allocating memory

  • runqhead: the dequeue number of the local run queue

  • runqtail: the queue number of the local run queue

  • runq: an array of local run queues, which can hold 256 G

  • gfree: G free list, save G instances that can be reused after becoming _Gdead

  • gcBgMarkWorker: worker function of background GC, if it exists M will execute it first

  • gcw: GC's local work queue, the details will be analyzed in the next article (GC article)

implementation of go

When using the go command to create a goroutine, go will compile the go command into a call to runtime.newproc. The structure of the stack is as follows:

The first argument is funcval + the length of extra arguments, the second argument is funcval, and the following are extra arguments passed to the function executed in the goroutine.
funcval is defined here, fn is a pointer to the machine code of the function.
The processing of runtime.newproc is as follows:

  • Calculate the address of the extra argument argp

  • Get the caller's address (return address) pc

  • call newproc1 using systemstack

systemstack will switch the current g to g0, and use the stack space of g0, then call the incoming function, and then switch back to the original g and the original stack space.
After switching to g0, it will pretend that the return address is mstart, so that the traceback can be stopped at mstart.
What is passed to systemstack here is a closure, and the address of the closure will be placed in the register rdx when it is called. For details, please refer to the analysis of closures above.

The processing of runtime.newproc1 is as follows:

  • Call getg to get the current g, it will be compiled to read the FS register (TLS), and g0 will be obtained here

  • Set the locks++ of m corresponding to g, prohibit preemption

  • get p owned by m

  • create a new g

    • First, call gfget to get g from p.gfree. If g has been recycled before, it can be reused here.

    • When it is not available, call malg to allocate a g, the initial stack space size is 2K

    • You need to set the state of g to aborted (_Gdead) first, so that gc will not scan the uninitialized stack of this g

  • copy the arguments to the stack of g

  • Copy the return address to the stack of g, where the return address is goexit, which means that goexit will be called after calling the target function

  • Set the scheduling data for g (sched)

    • Set sched.sp equal to the rsp address after parameter + return address

    • Set sched.pc equal to the address of the target function, see gostartcallfn and gostartcall

    • Set sched.g equal to g

  • Set the state of g to runnable (_Grunnable)

  • Call runqput to put g on the run queue

    • runqputslow will put half of the g in the local run queue into the global run queue, so that next time you can continue to use the fast local run queue

    • First, put g randomly into p.runnext, if it is put into runnext, it will join the g that was originally in runnext.

    • Then try to put g on P's "local run queue"

    • If the local run queue is full, call runqputslow to put g on the "global run queue"

  • If there is currently idle P, but no spin M(nmspinning is equal to 0), and the main function has been executed, wake up or create a new M

    • First swap nmspinning to 1, then continue after success, only one will continue if multiple threads execute wakep at the same time

    • call startm function

    • newm will create a new instance of m, the instance of m contains a g0, and then call newosproc to start a system thread

    • newosproc will call syscall clone to create a new thread

    • After the thread is created, TLS is set, the current g in TLS is set to g0, and then mstart is executed

    • Call pidleget to get a free P from the "free P list"

    • Call mget to get a free M from the "free M linked list"

    • If there is no free M, call newm to create a new M

    • Call notewakeup(&mp.park) to wake up the thread

    • This step is very important to ensure that there are currently enough M to run G. For details, please refer to the "Free M Linked List" above.

    • Wake up or create a new M will pass the wakep function

That's all for the process of creating goroutine s, let's see how M is scheduled.

Implementation of the scheduler

The mstart function is called when M starts, m0 is called after initialization, and the other m are called after the thread starts.
The processing of the mstart function is as follows:

  • Call getg to get the current g, here will get g0

  • If g does not allocate the stack, it will be allocated from the current stack space (system stack space), that is to say, g0 will use the system stack space

  • call mstart1 function

    • Call the gosave function to save the current state to the scheduling data of g0, and each subsequent scheduling will start from this stack address

    • call the asminit function, do nothing

    • Call the minit function to set the signal that the current thread can receive (signal)

    • Call the schedule function

After calling the schedule function, it enters the scheduling loop. The whole process can be simply summarized as:

schedule function acquisition g => [sleep when necessary] => [Continue to acquire after waking up] => execute function execution g => return to goexit => do it again schedule function

The processing of the schedule function is as follows:

  • If the current GC needs to stop the whole world (STW), call stopm to sleep the current M

  • If a function that needs to be run at a safe point (P.runSafePointFn) is specified in P owned by M, run it

  • Quickly obtain the G to be run, if one of the following processes is successfully obtained, it will not continue to obtain

    • If the current GC is marking the stage, check if there is a GC Worker to be run, and the GC Worker is also a G

    • For the sake of fairness, G is obtained from the global run queue every 61 schedules, (always obtaining G from the local run queue may cause G in the global run queue not to be run)

    • Get G from P's local run queue and call the runqget function

  • When the quick acquisition fails, call the findrunnable function to get the G to be run, which will block until the acquisition succeeds

    • Check again whether the current GC is in the marking phase, and then find out whether there is a GC Worker to be run, and the GC Worker is also a G

    • Double check if the current GC needs to stop the whole world, or if P specifies a function that needs to be run at a safe point, then jump to the top of the findrunnable and try again

    • Check again whether there is G in the global run queue, get it and return it

    • Release P owned by M, P will become idle (_Pidle) state

    • Add P to the "free P list"

    • Let M leave the spin state, the processing here is very important, refer to the "free M linked list" above

    • First reduce the global variable nmspinning representing the number of M in the current spin

    • Check all P's local runqueues again, let M re-spin if not empty, and jump to the top of the findrunnable to try again

    • Check again if there are GC Worker s to be run, if yes, let M re-enter the spin state, and jump to the top of findrunnable to try again

    • Check again if the network event reactor has G to run, where the call to netpoll will block until an event is received by an fd

    • If you still can't get G in the end, call stopm to sleep the current M

    • Retry after jumping to the top of findrunnable after waking up

    • Call runqsteal to try to steal half of G from other P's local runqueue

    • If the current GC needs to stop the whole world (STW), call stopm to sleep the current M

    • If a function that needs to be run at a safe point (P.runSafePointFn) is specified in P owned by M, run it

    • If there is a destructor to run then use "G to run the destructor"

    • Get G from P's local run queue and call the runqget function

    • Get G from the global run queue, call the globrunqget function, need to lock

    • Get G from the network event reactor, the function netpoll will get which fds are readable, writable or closed, and then return the G waiting for fd-related events

    • If G cannot be obtained, execute Work Stealing

    • If you still can't get G, you need to sleep M, the next step is to sleep

  • Successfully obtained a G to be run

  • Let M leave the spin state and call resetspinning, the processing here is different from the above

    • If there is currently idle P, but no spin M(nmspinning is equal to 0), wake up or create a new M

    • The above is to leave the spin state to sleep M, so all queues will be checked again and then sleep

    • The purpose of leaving the optional state here is to execute G, so it will check whether there is any free P, and if there is, it means that a new M can be opened to execute G.

  • If G asks to fall back to the specified M (such as runtime.main above)

    • Call the startlockedm function to give G and P to the M, and go to sleep by itself

    • Retry by jumping to the top of the schedule after waking up from sleep

  • Call the execute function to execute G

The processing of the execute function is as follows:

  • Call getg to get the current g

  • Change the status of G from running (_Grunnable) to running (_Grunning)

  • Set the stackguard of G, which can be expanded when the stack space is insufficient

  • Increase the number of schedules recorded in P (corresponding to the above priority to obtain a global run queue every 61 times)

  • set g.m.curg = g

  • set g.m = m

  • call the gogo function

    • This function will restore the value of each register according to the state saved in g.sched and continue to run g

    • First call the write barrier for g.sched.ctxt (GC mark pointer survives), ctxt generally saves a pointer to [function + parameter]

    • Set g in TLS to g.sched.g, which is g itself

    • Set the rsp register to g.sched.rsp

    • Set the rax register to g.sched.ret

    • Set rdx register to g.sched.ctxt (context)

    • Set the rbp register to g.sched.rbp

    • Clear the information saved in sched

    • Jump to g.sched.pc

    • Because the newproc1 function that created the goroutine earlier sets the return address to goexit, the goexit function will be called when the function returns after running.

g.sched.pc will point to the first machine instruction of the target function when G runs for the first time,
If G is preempted or waits for resources to go to sleep, the state will be saved to g.sched before going to sleep,
g.sched.pc will become the address that needs to continue to execute after waking up. The implementation of "save state" will be explained below.

After the target function is executed, the goexit function will be called, the goexit function will call the goexit1 function, and the goexit1 function will call the goexit0 function through mcall.
The mcall function is used to implement "save state", and the processing is as follows:

  • Set g.sched.pc equal to the current return address

  • Set g.sched.sp equal to the value of register rsp

  • Set g.sched.g equal to the current g

  • Set g.sched.bp equal to the value of register rbp

  • Toggle current g in TLS equal to m.g0

  • Set the register rsp equal to g0.sched.sp, use the stack space of g0

  • Set the first parameter to the original g

  • Set the rdx register to a pointer to the function address (context)

  • Calls the specified function, does not return

The mcall function saves the current running state to g.sched, then switches to the stack space of g0 and g0, and then calls the specified function.
The step of returning to the stack space of g0 is very important, because at this time g has been interrupted, and continuing to use the stack space of g and other M wake up this g will have catastrophic consequences.
After G is interrupted or ended, it will return to the stack space of g0 through mcall to continue scheduling. The saved state of mcall called from goexit is actually redundant, because G has ended.

The goexit1 function will call the goexit0 function through mcall. When the goexit0 function is called, it has returned to the stack space of g0. The processing is as follows:

  • Change the state of G from running (_Grunning) to aborted (_Gdead)

  • Empty members of G

  • Call the dropg function to disassociate between M and G

  • Call the gfput function to put G in the free list of P, which can be reused when creating G next time

  • Call the schedule function to continue scheduling

After G ends, return to the schedule function, which ends a scheduling loop.
Not only will G end restart scheduling, but G will be preempted or waiting for resources to be scheduled again. Let's continue to look at these two cases.

Implementation of preemption

Above I mentioned that runtime.main will create an extra M to run the sysmon function, and preemption is implemented in sysmon.
sysmon will enter an infinite loop, sleep for 20us in the first round, then double the sleep time each time, and finally sleep for 10ms in each round.
In sysmon, there are netpool (obtain fd events), retake (preemption), forcegc (force gc by time), scavenge heap (release redundant items in the free list to reduce memory usage) and other processing.

The retake function is responsible for handling preemption, and the process is:

  • enumerate all P

    • call the preemptone function

    • set g.preempt = true

    • set g.stackguard0 = stackPreempt

    • Call handoffp to disassociate between M and P

    • If P is in a system call (_Psyscall), and after a sysmon loop (20us~10ms), preempt this P

    • If P is running (_Prunning), and after a sysmon loop and G running time exceeds forcePreemptNS(10ms), preempt this P

Why can preemption be achieved by setting stackguard?
Because this value is used to check whether the current stack space is sufficient, the beginning of the go function will compare this value to determine whether the stack needs to be expanded.
stackPreempt is a special constant whose value will be greater than any stack address, and will trigger stack expansion when checked.

The stack expansion calls the morestack_noctxt function, the morestack_noctxt function clears the rdx register and calls the morestack function.
The morestack function will save the state of G to g.sched, switch to the stack space of g0 and g0, and then call the newstack function.
The newstack function judges that g.stackguard0 is equal to stackPreempt, and knows that this is triggered by preemption. At this time, it will check again whether to preempt:

  • If M is locked (there is P in the local variable of the function), skip this preemption and call the gogo function to continue running G

  • If M is allocating memory, skip this preemption and call the gogo function to continue running G

  • If M is set to not be preempted currently, skip this preemption and call the gogo function to continue running G

  • If the state of M is not running, skip this preemption and call the gogo function to continue running G

Even if this preemption fails, because g.preempt is equal to true, some code in the runtime will reset stackPreempt to retry the next preemption.
If it is judged that it can be preempted, then continue to judge whether it is caused by GC, if so, perform mark processing on the stack space of G (scan the root object) and continue to run,
If it is not caused by GC, call the gopreempt_m function to complete the preemption.

The gopreempt_m function will call the goschedImpl function. The process of the goschedImpl function is:

  • Change the status of G from running (_Grunnable) to pending (_Grunnable)

  • Call the dropg function to disassociate between M and G

  • Call globrunqput to put G on the global run queue

  • Call the schedule function to continue scheduling

Because the priority of the global run queue is relatively low, each M will re-acquire this G for execution after a period of time.
The preemption mechanism ensures that no G will run for a long time and cause other Gs to fail to run.

Implementation of channel

In the process of goroutine running, sometimes it is necessary to wait for resources, and channel is the most typical resource.
The data of the channel is defined here, and the key members are as follows:

  • qcount: the number of elements in the current queue

  • dataqsiz: The number of elements that the queue can hold, if it is 0, it means that this channel has no buffer

  • buf: the buffer of the queue, the structure is a circular queue

  • elemsize: the size of the element

  • closed: is it closed

  • elemtype: the type of the element, used when judging whether to call the write barrier

  • sendx: the sequence number of the sent element

  • recvx: the sequence number of the received element

  • recvq: The linked list of G currently waiting to receive data from the channel (the actual type is the linked list of sudog)

  • sendq: The linked list of G currently waiting to send data to the channel (the actual type is the linked list of sudog)

  • lock: the thread lock used when operating the channel

The runtime.chansend1 function is actually called to send data to the channel, and the chansend1 function calls the chansend function. The process is:

  • Check channel.recvq for G of waiting receivers

    • If sudog.elem is not equal to nil, call the sendDirect function to copy the element directly from the sender

    • The sudog.elem waiting to be received is a pointer to the memory of the receiving target. If the receiving target is _, then elem is nil, and copying can be omitted

    • waiting to be sent sudog.elem is a pointer to the source target's memory

    • Call goready to restore sender's G after copying

    • Change the state of G from waiting (_Gwaiting) to running (_Grunnable)

    • Put G on P's local run queue

    • If there is currently idle P, but no spin M(nmspinning is equal to 0), wake up or create a new M

    • Switch to g0 to call ready function, switch back after calling

    • If there is, it means that the channel has no buffer or the buffer is empty

    • call the send function

    • After getting the data from the sender and waking up G, it can return from chansend

  • Determine if the element can be put into the buffer

    • If the buffer has free space, put the element into the buffer and return from chansend

  • No buffer or buffer is full, sender's G needs to wait

    • call the gopark function

    • The mcall function is the same as described above, it will save the current state to g.sched, then switch to the stack space of g0 and g0 and execute the specified function

    • The park_m function first changes the state of G from running (_Grunning) to waiting (_Gwaiting)

    • Then call the dropg function to disassociate between M and G

    • Then call the incoming unlock function, the unlock function here will unlock the channel.lock

    • Finally, call the schedule function to continue scheduling

    • Call the park_m function through the mcall function

    • get the current g

    • Create a new sudog

    • set sudog.elem = pointer to send memory

    • set sudog.g = g

    • set sudog.c=channel

    • set g.waiting = sudog

    • put sudog into channel.sendq

    • Call the goparkunlock function

  • Recovering from here means that it has been sent successfully or the channel is closed

    • Check whether sudog.param is nil, if it is nil, the channel is closed, throw panic

    • else release sudog and return

The runtime.chanrecv1 function is actually called to receive data from the channel, and the chanrecv1 function calls the chanrecv function. The process is:

  • Check channel.sendq for pending sender's G

    • If there is no buffer, call the recvDirect function to copy the element directly to the receiver

    • If there is a buffer, it means the buffer is full

    • After copying, call goready to restore the receiver's G, and the processing is the same as above

    • Copy the next element in the queue to be dequeued directly to the receiver

    • Copy the sent element to the position in the queue where it was just dequeued

    • At this time, the buffer is still full, but the sending sequence number and the receiving sequence number will increase by 1

    • If there is, it means that the channel has no buffer or the buffer is full. These two cases need to be handled separately (in order to ensure the same order of entry and exit)

    • call the recv function

    • After handing over the data to the receiver and waking up G, you can return from chanrecv

  • Determine if an element can be retrieved from the buffer

    • If the buffer has an element, take the element directly and return it from chanrecv

  • No buffer or no elements in the buffer, the receiver's G needs to wait

    • get the current g

    • Create a new sudog

    • set sudog.elem = pointer to receive memory

    • set sudog.g = g

    • set sudog.c=channel

    • set g.waiting = sudog

    • put sudog into channel.recvq

    • Call the goparkunlock function, the same as above

  • Resuming from here means that it has been successfully received or the channel is closed

    • Check if sudog.param is nil, if it is nil it means the channel is closed

    • Unlike sending, receiving will not throw panic, and will notify the channel has been closed through the return value

    • release sudog and return

The closechan function is actually called to close the channel, and the process is:

  • set channel.closed = 1

  • enumerate channel.recvq, clear them sudog.elem, set sudog.param = nil

  • Enumerate channel.sendq, set sudog.elem = nil, set sudog.param = nil

  • Call the goready function to restore the G of all receivers and senders

Tags: Java Go data structure

Posted by behicthebuilder on Fri, 14 Oct 2022 08:48:36 +1030

Popular Tags

©2023 Programs Wiki