• Other bits and bobs: • Back propagate errors • Learning method (SGD, AdaGrad, RMSprop, L-‐BFGS…) • Errors (Cross Entropy, meansquare errors…) • Unique layers (ConvoluDon, RBM, etc …) • Mix and Match bits • All the above is just simple manipulaDon of matrix code
• map[coordinate]float64 • type sparseMat64 struct { d []float64 r []uint32 c []uint32 } • type sparseMat64 []*cell • type sparseMat64 [][MAXCOL]float64 type coordinate struct { r, c int } type cell struct { col int, val float64 }
• map[coordinate]float64 • type sparseMat64 struct { d []float64 r []uint32 c []uint32 } • type sparseMat64 []*cell • type sparseMat64 [][MAXCOL]float64 type coordinate struct { r, c int } type cell struct { col int, val float64 } Most flexible Most useful for Hadamard Fastest, good for small MAXCOL Fastest to build
chunkSize := len(mat) / threads costChan := make(chan *cost) for chunk := 0; chunk < threads; chunk++ { var start int = chunk * chunkSize var end int if start+chunkSize > len(mat) { end = len(mat) } else { end = start + chunkSize } go nn.costFn(mat[start:end], chunkSize, costChan) }
. // This loop took a helluva long time 226 4.93s 4.93s for _, a := range actives { 227 29.47s 29.47s layer2Grad[i][a] += delta * hiddenAct[a] 228 1.02mins 1.02mins hiddenActGrad[a] += delta * nn.weights2[i][a] 229 . . }
a scalar by a vector void avx_scale64(const double *s, const size_t n, double *x, double *z) { static const size_t elements = 4; // with double precision, the processor can only handle 4 at a time. Maybe switch to float32? const size_t end = n / elements; const __m256d vy = _mm256_broadcast_sd(s); // broadcast a scalar into vy // load x and z into AVX dedicated registers __m256d *vz = (__m256d *)z; __m256d *vx = (__m256d *)x; for (size_t i=0; i < end; ++i) { vz[i] = _mm256_mul_pd(vx[i], vy); } } */
int) []float64 { // pad it so that at most there will be 64 items extra -‐ worse comes to worst, there are a bunch of 0s to add or mul size := length * 64 ptr := C._mm_malloc((C.size_t)(size), 64) header := reflect.SliceHeader{ Data: uintptr(unsafe.Pointer(ptr)), Len: length, Cap: length, } goSlice := *(*[]float64)(unsafe.Pointer(&header)) return goSlice } // somewhere later defer C._mm_free(unsafe.Pointer(&vec[0]))
for i := range rows { // calculate delta omitted // This loop took a helluva long time for _, a := range actives { layer2Grad[i][a] += delta * hiddenAct[a] hiddenActGrad[a] += delta * nn.weights2[i][a] } }
TEXT ·∙scale64(SB), 7, $0 MOVSD s+0(FP), X0 MOVQ x_data+8(FP), SI // move the first element to the indexing register MOVQ x_len+24(FP), BP // move the length into BP // check that there is more than 4 values in the slice SUBQ $4, BP JL remainders // jump to remainder // manually broadcast s into X0's lower bits, which is used for SIMD MOVLHPS X0, X0 loop: // load the first two elements, and then multiply by s (which is in X0) MOVUPD (SI), X2 MULPD X0, X2 // load second two elements, then multiply by s MOVUPD 16(SI), X4 MULPD X0, X4 // move the values back to the original positions MOVUPD X2, (SI) MOVUPD X4, 16(SI) // now that we are done with the first 4 elements, update the pointers to the top of the next 4 elements ADDQ $32, SI SUBQ $4, BP // the len is now less 4 JGE loop remainders: ADDQ $4, BP // we subtracted earlier, remember??!!! JEQ done // if there are nothing left to process, we'll go to done remainingloop: // load element from x, and then multiply by s (which in X0) // TODO: check if there is a performance penalty on using SSE registers MOVSD (SI), X2 MULSD X0, X2 // save it back to the array MOVSD X2, (SI) // update the pointer ADDQ $8, SI DECQ BP JNE remainingloop done: RET
. testing: warning: no tests to run PASS BenchmarkGoScale64-‐8 1000000 1339 ns/op BenchmarkScale64-‐8 5000000 329 ns/op BenchmarkGoAdd64-‐8 1000000 1711 ns/op BenchmarkAdd64-‐8 1000000 1473 ns/op ok github.com/chewxy/testideas3 6.567s func GoAdd64(a, b []float64) { for i, v := range a{ a[i] = v + b[i] } } Not much difference
Be-er algorithms • PrecalculaDng acDvaDons and caching§ • Binary weights (no mulDplicaDon, only addiDon)* *very very very very very new. I'm sDll playing with it
network in Go is fairly simple. • Building a performant neural network in Go? Only embark with quesDonable sanity • Irony: deploying Theano is hard. But all my opDmizaDons make deploying this equally hard. • A default built in, well opDmized matrix type would be really really really sweet
chewxy
codelynx
rkaga
nrslib
kokuyouwind
penpeen
448jp
ahuglajbclajep
tasshi
florets1
masahiro_nishimi
inc2734
uyuki234
csswizardry
pwiseman
bkeepers
rasagy
joshbly
ammeep
destraynor
tmiket
addyosmani
sachag
brad_frost
afnizarnur
![@chewxy on Twi-er // func scale64(s float64, a []float64)](https://files.speakerdeck.com/presentations/ef112cdc8e95411897272b08170219a1/slide_38.jpg){kind=link}
